tabletalk 0.2.1__tar.gz → 0.5.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (132) hide show
  1. tabletalk-0.5.0/.github/workflows/db_provider_tests.yml +84 -0
  2. tabletalk-0.5.0/.github/workflows/publish.yml +96 -0
  3. {tabletalk-0.2.1 → tabletalk-0.5.0}/.gitignore +26 -0
  4. tabletalk-0.5.0/PKG-INFO +104 -0
  5. tabletalk-0.5.0/README.md +75 -0
  6. {tabletalk-0.2.1 → tabletalk-0.5.0}/docs/Makefile +1 -1
  7. tabletalk-0.5.0/docs/architecture.md +28 -0
  8. tabletalk-0.5.0/docs/commands.md +16 -0
  9. tabletalk-0.5.0/docs/conf.py +48 -0
  10. tabletalk-0.5.0/docs/configuration.md +48 -0
  11. tabletalk-0.5.0/docs/contributing.md +29 -0
  12. tabletalk-0.5.0/docs/evals.md +49 -0
  13. tabletalk-0.5.0/docs/getting-started.md +38 -0
  14. tabletalk-0.5.0/docs/index.md +14 -0
  15. tabletalk-0.5.0/examples/dbt-analytics/README.md +24 -0
  16. tabletalk-0.5.0/examples/dbt-analytics/agents/revenue.yaml +10 -0
  17. tabletalk-0.5.0/examples/dbt-analytics/dbt_project.yml +10 -0
  18. tabletalk-0.5.0/examples/dbt-analytics/evals/revenue.yaml +21 -0
  19. tabletalk-0.5.0/examples/dbt-analytics/models/marts/fct_orders.sql +2 -0
  20. tabletalk-0.5.0/examples/dbt-analytics/models/other/duplicate_orders.sql +2 -0
  21. tabletalk-0.5.0/examples/dbt-analytics/models/schema.yml +44 -0
  22. tabletalk-0.5.0/examples/dbt-analytics/models/sources.yml +19 -0
  23. tabletalk-0.5.0/examples/dbt-analytics/models/staging/disabled_orders.sql +2 -0
  24. tabletalk-0.5.0/examples/dbt-analytics/models/staging/ephemeral_order_keys.sql +2 -0
  25. tabletalk-0.5.0/examples/dbt-analytics/models/staging/stg_orders.sql +6 -0
  26. tabletalk-0.5.0/examples/dbt-analytics/package-lock.yml +4 -0
  27. tabletalk-0.5.0/examples/dbt-analytics/packages/shared_models/dbt_project.yml +4 -0
  28. tabletalk-0.5.0/examples/dbt-analytics/packages/shared_models/models/schema.yml +8 -0
  29. tabletalk-0.5.0/examples/dbt-analytics/packages/shared_models/models/shared_calendar.sql +1 -0
  30. tabletalk-0.5.0/examples/dbt-analytics/packages.yml +2 -0
  31. tabletalk-0.5.0/examples/dbt-analytics/profiles.yml +8 -0
  32. tabletalk-0.5.0/examples/dbt-analytics/seeds/raw_orders.csv +4 -0
  33. tabletalk-0.5.0/examples/dbt-analytics/tabletalk.yaml +11 -0
  34. tabletalk-0.5.0/examples/dbt-analytics/target/catalog.json +1 -0
  35. tabletalk-0.5.0/examples/dbt-analytics/target/manifest.json +1 -0
  36. {tabletalk-0.2.1 → tabletalk-0.5.0}/pyproject.toml +20 -26
  37. tabletalk-0.5.0/tabletalk/__init__.py +22 -0
  38. tabletalk-0.5.0/tabletalk/agents.py +279 -0
  39. tabletalk-0.5.0/tabletalk/authoring.py +106 -0
  40. tabletalk-0.5.0/tabletalk/cli.py +832 -0
  41. tabletalk-0.5.0/tabletalk/connections.py +166 -0
  42. tabletalk-0.5.0/tabletalk/evals/__init__.py +538 -0
  43. tabletalk-0.5.0/tabletalk/factories.py +114 -0
  44. tabletalk-0.5.0/tabletalk/interfaces.py +78 -0
  45. tabletalk-0.5.0/tabletalk/manifest.py +465 -0
  46. tabletalk-0.5.0/tabletalk/project.py +172 -0
  47. tabletalk-0.5.0/tabletalk/providers/duckdb_provider.py +26 -0
  48. tabletalk-0.5.0/tabletalk/providers/openai_provider.py +196 -0
  49. tabletalk-0.5.0/tabletalk/providers/snowflake_provider.py +46 -0
  50. tabletalk-0.5.0/tabletalk/providers/sqlite_provider.py +30 -0
  51. tabletalk-0.5.0/tabletalk/runtime/__init__.py +434 -0
  52. tabletalk-0.5.0/tabletalk/tests/test_connections.py +108 -0
  53. tabletalk-0.5.0/tabletalk/tests/test_dbt_native.py +900 -0
  54. tabletalk-0.5.0/tabletalk/traces.py +131 -0
  55. tabletalk-0.5.0/tabletalk/validation.py +199 -0
  56. tabletalk-0.5.0/uv.lock +2741 -0
  57. tabletalk-0.2.1/.github/workflows/db_provider_tests.yml +0 -57
  58. tabletalk-0.2.1/.github/workflows/publish.yml +0 -31
  59. tabletalk-0.2.1/PKG-INFO +0 -494
  60. tabletalk-0.2.1/README.md +0 -451
  61. tabletalk-0.2.1/contexts/default_context.yaml +0 -13
  62. tabletalk-0.2.1/docs/conf.py +0 -27
  63. tabletalk-0.2.1/docs/dev/contexts.rst +0 -45
  64. tabletalk-0.2.1/docs/dev/local_development.rst +0 -127
  65. tabletalk-0.2.1/docs/dev/providers.rst +0 -164
  66. tabletalk-0.2.1/docs/dev/querying.rst +0 -201
  67. tabletalk-0.2.1/docs/index.rst +0 -26
  68. tabletalk-0.2.1/examples/ecommerce/README.md +0 -281
  69. tabletalk-0.2.1/examples/ecommerce/agents/customer_analyst.yaml +0 -44
  70. tabletalk-0.2.1/examples/ecommerce/agents/inventory_manager.yaml +0 -34
  71. tabletalk-0.2.1/examples/ecommerce/agents/marketing_analyst.yaml +0 -35
  72. tabletalk-0.2.1/examples/ecommerce/agents/revenue_analyst.yaml +0 -39
  73. tabletalk-0.2.1/examples/ecommerce/contexts/customers.yaml +0 -30
  74. tabletalk-0.2.1/examples/ecommerce/contexts/inventory.yaml +0 -41
  75. tabletalk-0.2.1/examples/ecommerce/contexts/marketing.yaml +0 -39
  76. tabletalk-0.2.1/examples/ecommerce/contexts/sales.yaml +0 -53
  77. tabletalk-0.2.1/examples/ecommerce/dbt_project/dbt_project.yml +0 -16
  78. tabletalk-0.2.1/examples/ecommerce/dbt_project/models/marts/fct_orders.sql +0 -15
  79. tabletalk-0.2.1/examples/ecommerce/dbt_project/models/marts/fct_orders.yml +0 -17
  80. tabletalk-0.2.1/examples/ecommerce/dbt_project/models/sources.yml +0 -22
  81. tabletalk-0.2.1/examples/ecommerce/dbt_project/models/staging/stg_customers.sql +0 -10
  82. tabletalk-0.2.1/examples/ecommerce/dbt_project/models/staging/stg_orders.sql +0 -10
  83. tabletalk-0.2.1/examples/ecommerce/dbt_project/profiles.yml +0 -13
  84. tabletalk-0.2.1/examples/ecommerce/ecommerce.duckdb +0 -0
  85. tabletalk-0.2.1/examples/ecommerce/manifest/customers.txt +0 -5
  86. tabletalk-0.2.1/examples/ecommerce/manifest/inventory.txt +0 -7
  87. tabletalk-0.2.1/examples/ecommerce/manifest/marketing.txt +0 -7
  88. tabletalk-0.2.1/examples/ecommerce/manifest/sales.txt +0 -9
  89. tabletalk-0.2.1/examples/ecommerce/seed.py +0 -317
  90. tabletalk-0.2.1/examples/ecommerce/tabletalk.yaml +0 -51
  91. tabletalk-0.2.1/tabletalk/__init__.py +0 -27
  92. tabletalk-0.2.1/tabletalk/app.py +0 -385
  93. tabletalk-0.2.1/tabletalk/cli.py +0 -535
  94. tabletalk-0.2.1/tabletalk/factories.py +0 -155
  95. tabletalk-0.2.1/tabletalk/interfaces.py +0 -507
  96. tabletalk-0.2.1/tabletalk/profiles.py +0 -127
  97. tabletalk-0.2.1/tabletalk/providers/anthropic_provider.py +0 -59
  98. tabletalk-0.2.1/tabletalk/providers/azuresql_provider.py +0 -179
  99. tabletalk-0.2.1/tabletalk/providers/bigquery_provider.py +0 -98
  100. tabletalk-0.2.1/tabletalk/providers/duckdb_provider.py +0 -139
  101. tabletalk-0.2.1/tabletalk/providers/mysql_provider.py +0 -186
  102. tabletalk-0.2.1/tabletalk/providers/openai_provider.py +0 -51
  103. tabletalk-0.2.1/tabletalk/providers/postgres_provider.py +0 -164
  104. tabletalk-0.2.1/tabletalk/providers/snowflake_provider.py +0 -180
  105. tabletalk-0.2.1/tabletalk/providers/sqlite_provider.py +0 -84
  106. tabletalk-0.2.1/tabletalk/static/index.html +0 -1282
  107. tabletalk-0.2.1/tabletalk/static/logo.jpg +0 -0
  108. tabletalk-0.2.1/tabletalk/tests/conftest.py +0 -573
  109. tabletalk-0.2.1/tabletalk/tests/providers/__init__.py +0 -0
  110. tabletalk-0.2.1/tabletalk/tests/providers/test_azuresql.py +0 -61
  111. tabletalk-0.2.1/tabletalk/tests/providers/test_bigquery.py +0 -60
  112. tabletalk-0.2.1/tabletalk/tests/providers/test_duckdb.py +0 -377
  113. tabletalk-0.2.1/tabletalk/tests/providers/test_mysql.py +0 -129
  114. tabletalk-0.2.1/tabletalk/tests/providers/test_postgres.py +0 -114
  115. tabletalk-0.2.1/tabletalk/tests/providers/test_snowflake.py +0 -71
  116. tabletalk-0.2.1/tabletalk/tests/providers/test_sqlite.py +0 -364
  117. tabletalk-0.2.1/tabletalk/tests/test_app.py +0 -505
  118. tabletalk-0.2.1/tabletalk/tests/test_cli.py +0 -488
  119. tabletalk-0.2.1/tabletalk/tests/test_factories.py +0 -210
  120. tabletalk-0.2.1/tabletalk/tests/test_interfaces.py +0 -597
  121. tabletalk-0.2.1/tabletalk/tests/test_profiles.py +0 -378
  122. tabletalk-0.2.1/tabletalk/tests/test_utils.py +0 -218
  123. tabletalk-0.2.1/tabletalk/utils.py +0 -122
  124. tabletalk-0.2.1/tabletalk.yaml +0 -17
  125. tabletalk-0.2.1/uv.lock +0 -2416
  126. {tabletalk-0.2.1 → tabletalk-0.5.0}/.pre-commit-config.yaml +0 -0
  127. {tabletalk-0.2.1 → tabletalk-0.5.0}/.python-version +0 -0
  128. {tabletalk-0.2.1 → tabletalk-0.5.0}/LICENSE +0 -0
  129. {tabletalk-0.2.1 → tabletalk-0.5.0}/docs/make.bat +0 -0
  130. {tabletalk-0.2.1 → tabletalk-0.5.0}/tabletalk/main.py +0 -0
  131. {tabletalk-0.2.1 → tabletalk-0.5.0}/tabletalk/providers/__init__.py +0 -0
  132. {tabletalk-0.2.1 → tabletalk-0.5.0}/tabletalk/tests/__init__.py +0 -0
@@ -0,0 +1,84 @@
1
+ name: Test supported providers
2
+
3
+ on:
4
+ push:
5
+ branches: [main]
6
+ pull_request:
7
+ branches: [main]
8
+ workflow_dispatch:
9
+
10
+ permissions:
11
+ contents: read
12
+
13
+ jobs:
14
+ test:
15
+ name: Python ${{ matrix.python-version }}
16
+ runs-on: ubuntu-latest
17
+ strategy:
18
+ fail-fast: false
19
+ matrix:
20
+ python-version: ["3.10", "3.11", "3.12"]
21
+
22
+ steps:
23
+ - uses: actions/checkout@v6
24
+
25
+ - name: Install uv
26
+ uses: astral-sh/setup-uv@08807647e7069bb48b6ef5acd8ec9567f424441b # v8.1.0
27
+ with:
28
+ enable-cache: true
29
+ cache-suffix: py${{ matrix.python-version }}
30
+
31
+ - name: Set up Python
32
+ uses: actions/setup-python@v6
33
+ with:
34
+ python-version: ${{ matrix.python-version }}
35
+
36
+ - name: Install supported database extras
37
+ run: uv sync --all-extras
38
+
39
+ - name: Run tests
40
+ run: uv run pytest -q
41
+
42
+ - name: Run static checks
43
+ run: |
44
+ uv run ruff check tabletalk
45
+ uv run mypy tabletalk
46
+
47
+ package:
48
+ name: Verify PyPI distribution
49
+ runs-on: ubuntu-latest
50
+
51
+ steps:
52
+ - uses: actions/checkout@v6
53
+
54
+ - name: Install uv
55
+ uses: astral-sh/setup-uv@08807647e7069bb48b6ef5acd8ec9567f424441b # v8.1.0
56
+
57
+ - name: Set up Python
58
+ uses: actions/setup-python@v6
59
+ with:
60
+ python-version: "3.11"
61
+
62
+ - name: Build distribution
63
+ run: uv build
64
+
65
+ - name: Validate PyPI metadata
66
+ run: uvx twine check --strict dist/*
67
+
68
+ - name: Verify wheel contents and installation
69
+ run: |
70
+ python - <<'PY'
71
+ from pathlib import Path
72
+ from zipfile import ZipFile
73
+
74
+ wheel = next(Path("dist").glob("*.whl"))
75
+ with ZipFile(wheel) as archive:
76
+ names = archive.namelist()
77
+ assert not any("/tests/" in name for name in names)
78
+ assert not any(".data/data/tabletalk/static/" in name for name in names)
79
+ assert "tabletalk/static/index.html" in names
80
+ PY
81
+ uv venv .package-test
82
+ uv pip install --python .package-test/bin/python dist/*.whl
83
+ .package-test/bin/tabletalk --version
84
+ .package-test/bin/python -c "from tabletalk import Agent, EvalSuite, Project"
@@ -0,0 +1,96 @@
1
+ name: Publish to PyPI
2
+
3
+ on:
4
+ release:
5
+ types: [published]
6
+
7
+ permissions:
8
+ contents: read
9
+
10
+ jobs:
11
+ build:
12
+ name: Build and verify distribution
13
+ runs-on: ubuntu-latest
14
+
15
+ steps:
16
+ - uses: actions/checkout@v6
17
+
18
+ - name: Install uv
19
+ uses: astral-sh/setup-uv@08807647e7069bb48b6ef5acd8ec9567f424441b # v8.1.0
20
+ with:
21
+ enable-cache: true
22
+
23
+ - name: Set up Python
24
+ uses: actions/setup-python@v6
25
+ with:
26
+ python-version: "3.11"
27
+
28
+ - name: Verify release tag matches package version
29
+ run: |
30
+ PACKAGE_VERSION="$(python -c 'import tomllib; print(tomllib.load(open("pyproject.toml", "rb"))["project"]["version"])')"
31
+ test "${GITHUB_REF_NAME#v}" = "$PACKAGE_VERSION"
32
+
33
+ - name: Build wheel and source distribution
34
+ run: uv build
35
+
36
+ - name: Validate PyPI metadata
37
+ run: uvx twine check --strict dist/*
38
+
39
+ - name: Inspect wheel contents
40
+ run: |
41
+ python - <<'PY'
42
+ from pathlib import Path
43
+ from zipfile import ZipFile
44
+
45
+ wheel = next(Path("dist").glob("*.whl"))
46
+ with ZipFile(wheel) as archive:
47
+ names = archive.namelist()
48
+ assert not any("/tests/" in name for name in names)
49
+ assert not any("tabletalk/static/" in name for name in names)
50
+ assert "tabletalk/app.py" not in names
51
+ for required in (
52
+ "tabletalk/agents.py",
53
+ "tabletalk/evals/__init__.py",
54
+ "tabletalk/manifest.py",
55
+ "tabletalk/project.py",
56
+ "tabletalk/runtime/__init__.py",
57
+ "tabletalk/validation.py",
58
+ ):
59
+ assert required in names, f"missing wheel module: {required}"
60
+ PY
61
+
62
+ - name: Verify wheel installation
63
+ run: |
64
+ uv venv .package-test
65
+ uv pip install --python .package-test/bin/python dist/*.whl
66
+ .package-test/bin/tabletalk --version
67
+ .package-test/bin/python -c "from tabletalk import Agent, EvalSuite, Project"
68
+
69
+ - name: Upload verified distributions
70
+ uses: actions/upload-artifact@v7
71
+ with:
72
+ name: python-package-distributions
73
+ path: dist/
74
+ if-no-files-found: error
75
+
76
+ publish:
77
+ name: Publish verified distribution
78
+ needs: build
79
+ runs-on: ubuntu-latest
80
+ environment:
81
+ name: pypi
82
+ url: https://pypi.org/p/tabletalk
83
+ permissions:
84
+ id-token: write
85
+
86
+ steps:
87
+ - name: Download verified distributions
88
+ uses: actions/download-artifact@v8
89
+ with:
90
+ name: python-package-distributions
91
+ path: dist/
92
+
93
+ - name: Publish to PyPI using trusted publishing
94
+ uses: pypa/gh-action-pypi-publish@release/v1
95
+ with:
96
+ packages-dir: dist/
@@ -65,8 +65,28 @@ user_settings.json
65
65
 
66
66
  # Database
67
67
  *.db
68
+ *.duckdb
69
+ *.duckdb.wal
70
+ examples/ecommerce/evals/fixtures/*.duckdb
68
71
  __pycache__
69
72
 
73
+ # TableTalk generated output and per-user runtime state
74
+ .tabletalk/
75
+ /examples/*/manifest/
76
+ .tabletalk_history.jsonl
77
+ .tabletalk_favorites.json
78
+ .tabletalk_audit.jsonl
79
+ .tabletalk_schedules.json
80
+ .tabletalk_history/
81
+ .tabletalk_memory/
82
+
83
+ # dbt generated and per-user state
84
+ .user.yml
85
+ dbt_packages/
86
+ logs/
87
+ /examples/*/dbt_project/dbt_packages/
88
+ /examples/*/dbt_project/logs/
89
+
70
90
  # PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
71
91
  __pypackages__/
72
92
 
@@ -128,6 +148,12 @@ docs/_build/
128
148
  .pybuilder/
129
149
  target/
130
150
 
151
+ # The reference project's manifest is a dbt-generated acceptance fixture.
152
+ !examples/dbt-analytics/target/
153
+ examples/dbt-analytics/target/*
154
+ !examples/dbt-analytics/target/manifest.json
155
+ !examples/dbt-analytics/target/catalog.json
156
+
131
157
  # Jupyter Notebook
132
158
  .ipynb_checkpoints
133
159
 
@@ -0,0 +1,104 @@
1
+ Metadata-Version: 2.4
2
+ Name: tabletalk
3
+ Version: 0.5.0
4
+ Summary: Evaluation and observability for natural-language agents over dbt projects
5
+ Project-URL: Homepage, https://github.com/wtbates99/tabletalk
6
+ Project-URL: Repository, https://github.com/wtbates99/tabletalk
7
+ Author-email: william bates <wtbates99@gmail.com>
8
+ License: CC BY-NC 4.0
9
+ License-File: LICENSE
10
+ Keywords: ai,database,dbt,llm,natural-language,sql
11
+ Classifier: Development Status :: 3 - Alpha
12
+ Classifier: Programming Language :: Python :: 3.10
13
+ Classifier: Programming Language :: Python :: 3.11
14
+ Classifier: Programming Language :: Python :: 3.12
15
+ Requires-Python: >=3.10
16
+ Requires-Dist: click>=8.0
17
+ Requires-Dist: openai>=1.0.0
18
+ Requires-Dist: pyyaml>=6.0
19
+ Requires-Dist: rich>=13.0
20
+ Requires-Dist: sqlglot>=25.0
21
+ Provides-Extra: all
22
+ Requires-Dist: duckdb>=0.9; extra == 'all'
23
+ Requires-Dist: snowflake-connector-python>=3.0; extra == 'all'
24
+ Provides-Extra: duckdb
25
+ Requires-Dist: duckdb>=0.9; extra == 'duckdb'
26
+ Provides-Extra: snowflake
27
+ Requires-Dist: snowflake-connector-python>=3.0; extra == 'snowflake'
28
+ Description-Content-Type: text/markdown
29
+
30
+ # TableTalk
31
+
32
+ TableTalk is an evaluation and observability framework for natural-language agents built on existing
33
+ dbt projects. `manifest.json` is the complete authority for what an agent may query; warehouse
34
+ connections are read-only execution adapters for those dbt relations.
35
+
36
+ ## Start from a dbt project
37
+
38
+ ```console
39
+ cd my-dbt-project
40
+ dbt parse
41
+ dbt docs generate # recommended: adds physical warehouse types
42
+ tabletalk init
43
+ tabletalk agent create
44
+ tabletalk eval create revenue
45
+ tabletalk eval run revenue
46
+ tabletalk ask revenue "What was recognized revenue last month?"
47
+ ```
48
+
49
+ `tabletalk init` finds `dbt_project.yml` and `target/manifest.json`, resolves a supported SQLite,
50
+ DuckDB, or Snowflake target from the dbt profile, and writes only non-secret TableTalk settings.
51
+ Agent creation asks the user to choose a displayed dbt group, tag, model, path, package, or source and
52
+ whether to include lineage. It previews the exact resolved models, descriptions, catalog types, tests,
53
+ constraints, and lineage before writing a small selector-based resource. Raw selectors remain available
54
+ for automation.
55
+
56
+ ```yaml
57
+ name: revenue
58
+ description: Answers questions about recognized revenue.
59
+ select:
60
+ - group:finance
61
+ - tag:revenue
62
+ exclude:
63
+ - model:customer_sensitive
64
+ instructions:
65
+ - Use recognized revenue unless the user explicitly requests bookings.
66
+ sample_questions:
67
+ - What was recognized revenue last month?
68
+ ```
69
+
70
+ Descriptions, columns, tests, constraints, ownership, access, tags, and lineage stay in dbt. They are
71
+ never duplicated into an agent file.
72
+
73
+ ## Correctness and provenance
74
+
75
+ Live questions and eval cases call the exact same `Runtime.answer` path. Before execution, TableTalk
76
+ parses generated SQL and requires one read-only query, in-scope manifest relations, known columns,
77
+ explicit join conditions, row and timeout limits, and explicit sensitive-data permission. It derives
78
+ used dbt nodes and columns from that parsed SQL—not from model output.
79
+
80
+ Every result includes the answer, interpretation, assumptions, generated and executed SQL, dbt nodes,
81
+ columns, relevant test health, bounded evidence, evidence-linked claims, verification outcomes,
82
+ manifest and agent fingerprints, model and warehouse identity, latency, and token usage. Terminal and
83
+ web views expose “How this answer was formed” without separate flags.
84
+
85
+ Eval correctness is determined by execution, scope, model/column expectations, result equality,
86
+ reference-query matching, shape/count assertions, tolerance, and claim evidence. Eval creation uses the
87
+ reviewed SQL as the default changing-data reference, runs it immediately, and writes a verification
88
+ record. A live answer is labeled `VERIFIED` only when its exact normalized question matches an approved
89
+ eval case and all hard checks pass; otherwise it is visibly `UNVERIFIED` and exits nonzero.
90
+
91
+ See the dbt-generated [reference project](examples/dbt-analytics/README.md) and the
92
+ [workflow guide](docs/getting-started.md).
93
+
94
+ ## Commands
95
+
96
+ - `tabletalk init`
97
+ - `tabletalk agent create|list|show`
98
+ - `tabletalk eval create|run`
99
+ - `tabletalk ask`
100
+ - `tabletalk doctor`
101
+
102
+ There is no compile/plan/apply lifecycle or applied-artifact authorization state. Source changes are
103
+ active immediately and reproducibility records live under `.tabletalk/runs` and
104
+ `.tabletalk/eval-results`.
@@ -0,0 +1,75 @@
1
+ # TableTalk
2
+
3
+ TableTalk is an evaluation and observability framework for natural-language agents built on existing
4
+ dbt projects. `manifest.json` is the complete authority for what an agent may query; warehouse
5
+ connections are read-only execution adapters for those dbt relations.
6
+
7
+ ## Start from a dbt project
8
+
9
+ ```console
10
+ cd my-dbt-project
11
+ dbt parse
12
+ dbt docs generate # recommended: adds physical warehouse types
13
+ tabletalk init
14
+ tabletalk agent create
15
+ tabletalk eval create revenue
16
+ tabletalk eval run revenue
17
+ tabletalk ask revenue "What was recognized revenue last month?"
18
+ ```
19
+
20
+ `tabletalk init` finds `dbt_project.yml` and `target/manifest.json`, resolves a supported SQLite,
21
+ DuckDB, or Snowflake target from the dbt profile, and writes only non-secret TableTalk settings.
22
+ Agent creation asks the user to choose a displayed dbt group, tag, model, path, package, or source and
23
+ whether to include lineage. It previews the exact resolved models, descriptions, catalog types, tests,
24
+ constraints, and lineage before writing a small selector-based resource. Raw selectors remain available
25
+ for automation.
26
+
27
+ ```yaml
28
+ name: revenue
29
+ description: Answers questions about recognized revenue.
30
+ select:
31
+ - group:finance
32
+ - tag:revenue
33
+ exclude:
34
+ - model:customer_sensitive
35
+ instructions:
36
+ - Use recognized revenue unless the user explicitly requests bookings.
37
+ sample_questions:
38
+ - What was recognized revenue last month?
39
+ ```
40
+
41
+ Descriptions, columns, tests, constraints, ownership, access, tags, and lineage stay in dbt. They are
42
+ never duplicated into an agent file.
43
+
44
+ ## Correctness and provenance
45
+
46
+ Live questions and eval cases call the exact same `Runtime.answer` path. Before execution, TableTalk
47
+ parses generated SQL and requires one read-only query, in-scope manifest relations, known columns,
48
+ explicit join conditions, row and timeout limits, and explicit sensitive-data permission. It derives
49
+ used dbt nodes and columns from that parsed SQL—not from model output.
50
+
51
+ Every result includes the answer, interpretation, assumptions, generated and executed SQL, dbt nodes,
52
+ columns, relevant test health, bounded evidence, evidence-linked claims, verification outcomes,
53
+ manifest and agent fingerprints, model and warehouse identity, latency, and token usage. Terminal and
54
+ web views expose “How this answer was formed” without separate flags.
55
+
56
+ Eval correctness is determined by execution, scope, model/column expectations, result equality,
57
+ reference-query matching, shape/count assertions, tolerance, and claim evidence. Eval creation uses the
58
+ reviewed SQL as the default changing-data reference, runs it immediately, and writes a verification
59
+ record. A live answer is labeled `VERIFIED` only when its exact normalized question matches an approved
60
+ eval case and all hard checks pass; otherwise it is visibly `UNVERIFIED` and exits nonzero.
61
+
62
+ See the dbt-generated [reference project](examples/dbt-analytics/README.md) and the
63
+ [workflow guide](docs/getting-started.md).
64
+
65
+ ## Commands
66
+
67
+ - `tabletalk init`
68
+ - `tabletalk agent create|list|show`
69
+ - `tabletalk eval create|run`
70
+ - `tabletalk ask`
71
+ - `tabletalk doctor`
72
+
73
+ There is no compile/plan/apply lifecycle or applied-artifact authorization state. Source changes are
74
+ active immediately and reproducibility records live under `.tabletalk/runs` and
75
+ `.tabletalk/eval-results`.
@@ -4,7 +4,7 @@
4
4
  # You can set these variables from the command line, and also
5
5
  # from the environment for the first two.
6
6
  SPHINXOPTS ?=
7
- SPHINXBUILD ?= sphinx-build
7
+ SPHINXBUILD ?= ../.venv/bin/sphinx-build
8
8
  SOURCEDIR = .
9
9
  BUILDDIR = _build
10
10
 
@@ -0,0 +1,28 @@
1
+ # Runtime safety and architecture
2
+
3
+ Every live and eval question follows one pipeline:
4
+
5
+ ```text
6
+ manifest → resolved agent scope → interpretation/SQL → AST validation
7
+ → read-only connector → evidence rows → answer/claims → trace verification
8
+ ```
9
+
10
+ `manifest.py` owns artifact normalization, selection, graph navigation, and fingerprints. `agents.py`
11
+ contains only selector resources and resolved in-memory scopes. `validation.py` parses SQL and derives
12
+ actual model/column use. `connections.py` resolves the selected dbt target and exposes only read-only
13
+ query execution. `runtime` constructs the answer and shared trace. `evals` calls that exact runtime and
14
+ adds deterministic comparisons. `traces.py` persists one schema for live and eval records.
15
+
16
+ Validation rejects multiple statements, writes/commands, forbidden external-read functions,
17
+ out-of-scope or ambiguous relations, unknown columns, unconditioned joins, excessive limits, and
18
+ unapproved sensitive resources. Connector permissions remain the final security boundary.
19
+
20
+ Lineage proves dependency, not semantic join correctness. TableTalk never labels a join verified merely
21
+ because two nodes share a lineage edge. Safe join evidence belongs in dbt constraints, relationship
22
+ tests, explicit `meta`, and passing eval coverage.
23
+
24
+ Post-execution verification requires successful execution and validates every claim’s row/column
25
+ evidence. Numeric claims must appear in cited result cells. Correctness additionally requires an exact
26
+ approved eval-question match and passing deterministic comparisons. Reported models and columns always
27
+ come from the validated AST. Trace records hold fingerprints and identities, not credentials or
28
+ deployment state.
@@ -0,0 +1,16 @@
1
+ # Commands
2
+
3
+ - `tabletalk init`: initialize from `dbt_project.yml`, a parsed manifest, and a dbt target.
4
+ - `tabletalk agent create`: guided selector preview and agent authoring.
5
+ - `tabletalk agent list`: list source agents and resolved model counts.
6
+ - `tabletalk agent show NAME`: show source, resolved node IDs, fingerprint, and warnings.
7
+ - `tabletalk eval create NAME`: execute and approve a question/reference case.
8
+ - `tabletalk eval run [NAME] [--case CASE] [--trials N]`: run deterministic hard-gate evals and
9
+ optionally override the suite's independent-trial count.
10
+ - `tabletalk ask NAME QUESTION`: answer (quoted or unquoted), show provenance, and require passing exact eval coverage for a
11
+ `VERIFIED` status.
12
+ - `tabletalk doctor`: fail on artifact, target, connectivity, selector, or eval-coverage blockers and
13
+ report incomplete dbt descriptions as non-blocking metadata warnings.
14
+
15
+ `compile`, `plan`, `apply`, `connect`, `discover`, `connections`, and registry-style `agents` commands
16
+ were removed. A source agent is active immediately and asking never depends on applied state.
@@ -0,0 +1,48 @@
1
+ # Configuration file for the Sphinx documentation builder.
2
+ # https://www.sphinx-doc.org/en/master/usage/configuration.html
3
+
4
+ # -- Project information -------------------------------------------------------
5
+
6
+ project = "tabletalk"
7
+ copyright = "2025, william bates"
8
+ author = "william bates"
9
+ release = "0.5.0"
10
+
11
+ # -- General configuration -----------------------------------------------------
12
+
13
+ extensions = [
14
+ "myst_parser",
15
+ ]
16
+
17
+ # MyST parser settings — enable common markdown extensions
18
+ myst_enable_extensions = [
19
+ "colon_fence", # ::: fenced directives
20
+ "deflist", # definition lists
21
+ "tasklist", # - [ ] checkboxes
22
+ ]
23
+
24
+ # Auto-generate anchors for headings up to level 3 (enables #anchor-name links)
25
+ myst_heading_anchors = 3
26
+
27
+ # Use index.md as the root document
28
+ root_doc = "index"
29
+
30
+ source_suffix = {
31
+ ".md": "markdown",
32
+ }
33
+
34
+ templates_path = ["_templates"]
35
+ exclude_patterns = ["_build", "Thumbs.db", ".DS_Store"]
36
+
37
+ # -- Options for HTML output ---------------------------------------------------
38
+
39
+ html_theme = "alabaster"
40
+ html_static_path = []
41
+
42
+ html_theme_options = {
43
+ "description": "dbt-native evaluation and observability for natural-language agents",
44
+ "github_user": "wtbates99",
45
+ "github_repo": "tabletalk",
46
+ "github_button": True,
47
+ "fixed_sidebar": True,
48
+ }
@@ -0,0 +1,48 @@
1
+ # Configuration and resources
2
+
3
+ `tabletalk.yaml` is intentionally small:
4
+
5
+ ```yaml
6
+ dbt:
7
+ project_dir: .
8
+ manifest: target/manifest.json
9
+ catalog: target/catalog.json
10
+ run_results: target/run_results.json
11
+ target: dev
12
+
13
+ llm:
14
+ provider: ollama
15
+ model: gemma4:31b-cloud
16
+ base_url: http://localhost:11434/v1
17
+ ```
18
+
19
+ Commands first look for `tabletalk.yaml` in the current directory, then for
20
+ `tabletalk/tabletalk.yaml`. This keeps TableTalk in its own repository folder without requiring a
21
+ `--project-folder` option on every command.
22
+
23
+ `project_dir`, `manifest`, and optional `profiles_dir` paths are relative to `tabletalk.yaml`. The dbt
24
+ target remains in `profiles.yml`; environment variables are resolved only at runtime. Supported
25
+ adapters are SQLite, DuckDB, and Snowflake, all opened or governed as read-only.
26
+
27
+ Agents support only `group:`, `tag:`, `model:`, `source:`, `path:`, and `package:` selectors plus exclusions.
28
+ `include_parents` and `include_children` are explicit booleans. Ephemeral and disabled models are not
29
+ queryable. Sources require an explicit `source:SOURCE.TABLE` selector; lineage expansion never grants
30
+ source access. Physical relation names must resolve to one dbt unique ID in the agent scope.
31
+
32
+ `instructions` documents business rules the manifest cannot express, while `reject_if_contains`
33
+ provides a small deterministic guardrail for known concepts that the selected dbt resources cannot
34
+ answer. Rejections happen before an LLM call and should be used sparingly for unambiguous missing-data
35
+ boundaries, not as a general intent router.
36
+
37
+ Optional `catalog.json` enriches physical types and statistics; optional `run_results.json` adds recent
38
+ dbt-test health. Neither can add queryable resources. The manifest fingerprint changes with any
39
+ manifest content change and is included in every run; the catalog has its own fingerprint. `tabletalk
40
+ init` detects both artifacts beside the manifest automatically.
41
+
42
+ The model prompt receives each selected resource's relation, description, physical catalog types,
43
+ column descriptions, tests, constraints, owner, access, materialization, package, explicit join
44
+ metadata, and upstream/downstream lineage. Lineage is provenance, never automatic join permission.
45
+
46
+ Sensitive models use `meta: {sensitive: true}`; sensitive columns use column-level
47
+ `meta: {sensitive: true}` or model `meta.sensitive_columns`. The agent must explicitly list allowed dbt
48
+ unique IDs or column names under `allow_sensitive`.
@@ -0,0 +1,29 @@
1
+ # Contributing
2
+
3
+ TableTalk requires Python 3.10+ and uses `uv`:
4
+
5
+ ```console
6
+ uv sync --all-extras
7
+ uv run pytest
8
+ uv run ruff check .
9
+ uv run mypy tabletalk
10
+ ```
11
+
12
+ The acceptance fixture is a real dbt project under `examples/dbt-analytics`. When its dbt resources
13
+ change, regenerate the checked-in artifact instead of hand-editing it:
14
+
15
+ ```console
16
+ cd examples/dbt-analytics
17
+ uv run dbt deps
18
+ uv run dbt parse --profiles-dir . --no-partial-parse
19
+ uv run dbt docs generate --profiles-dir .
20
+ ```
21
+
22
+ Keep these boundaries intact: manifest selection is the only source of query scope; database
23
+ introspection cannot add resources; all live and eval questions use `Runtime.answer`; model and column
24
+ usage comes from parsed SQL; result comparison is the hard correctness gate; and persisted records are
25
+ observability evidence, never deployment authorization.
26
+
27
+ Tests should cover manifest selection, SQL safety/scope, connectors, deterministic comparisons,
28
+ evidence-linked traces, and the complete dbt → agent → eval → ask journey. Snowflake behavior should use
29
+ a mocked connector contract unless a test is explicitly marked for live credentials.
@@ -0,0 +1,49 @@
1
+ # Execution-based evals
2
+
3
+ Suites may declare `kind: regression|capability`, a human-readable `description`, and `trials: 1..20`.
4
+ Regression suites are expected to remain nearly perfect; capability suites hold harder tasks with room
5
+ to improve. Repeated trials are persisted separately, all must pass for the command to succeed, and the
6
+ terminal reports aggregate trial pass rate.
7
+
8
+ Create evals interactively with `tabletalk eval create AGENT`. The default is the generated SQL the
9
+ user just reviewed, so changing warehouse data is compared by executing candidate and golden queries
10
+ against the same snapshot. The proposed case runs immediately and automated authoring refuses to save
11
+ an already-failing case. Literal rows remain available for stable fixtures and CI.
12
+
13
+ ```yaml
14
+ name: revenue-regression
15
+ agent: revenue
16
+ cases:
17
+ - name: july-revenue
18
+ question: What was recognized revenue in July 2026?
19
+ expect:
20
+ reference_sql: |
21
+ select sum(recognized_revenue) as recognized_revenue
22
+ from {{ ref('fct_orders') }}
23
+ where order_date >= '2026-07-01'
24
+ and order_date < '2026-08-01'
25
+ result:
26
+ comparison: scalar
27
+ tolerance: 0.01
28
+ models:
29
+ required: [model.analytics.fct_orders]
30
+ columns:
31
+ required: [recognized_revenue, order_date]
32
+ ```
33
+
34
+ Result modes are `scalar`, `ordered`, `ordered_values`, `unordered`, and `keyed`; keyed comparisons
35
+ require `keys`. `ordered_values` deliberately ignores presentation-only aliases while preserving row
36
+ and value order.
37
+ Expectations may also include literal `value` or `rows`, `row_count`, exact result `columns`, forbidden
38
+ models/columns, `allow_extra_columns: true` for harmless additional evidence fields, and
39
+ `outcome: ambiguity|rejection`. Reference SQL supports only dbt `ref()` templating,
40
+ is parsed with the same read-only scope validator, and executes against the same warehouse snapshot.
41
+
42
+ Hard gates are execution, SQL safety/scope, expected models/columns, expected rejection, result
43
+ comparison and tolerance, shape/count, and evidence-linked claims. Latency and token usage are recorded
44
+ but informational. Empty literal row sets are enforced rather than treated as a missing expectation.
45
+ Results are written to `.tabletalk/eval-results/AGENT` and failures exit with code 3.
46
+
47
+ Live correctness uses exact normalized question matching, not semantic guesswork. A covered question is
48
+ `VERIFIED` only when every hard gate passes. Uncovered questions still return their inspectable trace,
49
+ but are labeled `UNVERIFIED` and exit with code 4.
@@ -0,0 +1,38 @@
1
+ # Get started
2
+
3
+ TableTalk starts with a parsed, runnable dbt project. It does not discover arbitrary database tables.
4
+
5
+ ```console
6
+ cd my-dbt-project
7
+ dbt parse
8
+ dbt docs generate
9
+ tabletalk init
10
+ ```
11
+
12
+ Initialization finds the dbt project and manifest, reads the project profile, asks for a target, checks
13
+ that its adapter is SQLite, DuckDB, or Snowflake, summarizes dbt version/models/groups/tags, and writes
14
+ `tabletalk.yaml`. Credentials stay in environment variables and `profiles.yml`.
15
+
16
+ Create an agent from dbt selectors:
17
+
18
+ ```console
19
+ tabletalk agent create
20
+ ```
21
+
22
+ Choose a displayed dbt group, tag, model, path, package, or source, then choose whether to include its
23
+ lineage. The guided preview shows exact resources, catalog types, tests, constraints, and metadata gaps.
24
+ Create the first eval when prompted. TableTalk shows the interpretation and SQL before read-only
25
+ execution, uses that reviewed SQL as the default golden query, verifies it immediately, and saves both
26
+ the case and result record.
27
+
28
+ ```console
29
+ tabletalk eval run revenue
30
+ tabletalk ask revenue "What was recognized revenue last month?"
31
+ ```
32
+
33
+ Both commands use the same runtime. `ask` prints `VERIFIED` only for an exact normalized question match
34
+ against an approved, passing eval; new or changed questions print `UNVERIFIED` and exit nonzero. Run
35
+ `tabletalk doctor` in CI or locally to detect missing/stale
36
+ artifacts, unsupported targets, connectivity failures, broken selectors, metadata gaps, and uncovered
37
+ agents. Metadata gaps are warnings; conditions that prevent safe or verified execution fail the command.
38
+ The complete runnable journey is in `examples/dbt-analytics`.