duckcheck 0.4.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- duckcheck-0.4.0/.github/workflows/ci.yml +11 -0
- duckcheck-0.4.0/.gitignore +5 -0
- duckcheck-0.4.0/CHANGELOG.md +28 -0
- duckcheck-0.4.0/CONTRIBUTING.md +29 -0
- duckcheck-0.4.0/LICENSE +21 -0
- duckcheck-0.4.0/PKG-INFO +145 -0
- duckcheck-0.4.0/README.md +121 -0
- duckcheck-0.4.0/SECURITY.md +14 -0
- duckcheck-0.4.0/compose.yaml +11 -0
- duckcheck-0.4.0/docker/Dockerfile +8 -0
- duckcheck-0.4.0/examples/checks.yaml +22 -0
- duckcheck-0.4.0/examples/clean.csv +4 -0
- duckcheck-0.4.0/examples/clean.yaml +19 -0
- duckcheck-0.4.0/examples/sample.csv +5 -0
- duckcheck-0.4.0/plans/README.md +6 -0
- duckcheck-0.4.0/plans/v0.1-completed-foundation.md +200 -0
- duckcheck-0.4.0/plans/v0.2-next-phase-roadmap.md +356 -0
- duckcheck-0.4.0/pyproject.toml +46 -0
- duckcheck-0.4.0/src/duckcheck/__init__.py +3 -0
- duckcheck-0.4.0/src/duckcheck/baseline.py +36 -0
- duckcheck-0.4.0/src/duckcheck/cli.py +114 -0
- duckcheck-0.4.0/src/duckcheck/runner.py +352 -0
- duckcheck-0.4.0/src/duckcheck/spec.py +26 -0
- duckcheck-0.4.0/tests/test_runner.py +120 -0
|
@@ -0,0 +1,28 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
## [0.4.0] - 2026-09-14
|
|
4
|
+
|
|
5
|
+
### Added
|
|
6
|
+
- `custom_sql` `expect` operators: `0`, `=N`, `>N`, `<N`, `>=N`, `<=N` (default `0`)
|
|
7
|
+
- `duckcheck run --format json` for machine-readable CI output
|
|
8
|
+
|
|
9
|
+
## [0.3.0] - 2026-08-19
|
|
10
|
+
|
|
11
|
+
### Added
|
|
12
|
+
- SQLite sources (`sqlite://path` or `.db`) via DuckDB ATTACH
|
|
13
|
+
- `row_count_delta` checks with `duckcheck baseline update`
|
|
14
|
+
- `DUCKCHECK_NOW` to freeze freshness comparisons in tests/CI
|
|
15
|
+
|
|
16
|
+
## [0.2.0] - 2026-08-19
|
|
17
|
+
|
|
18
|
+
### Added
|
|
19
|
+
- Check types: `custom_sql`, `freshness`, `row_count`
|
|
20
|
+
- `--junit` XML output and `--source-table` override
|
|
21
|
+
- `${ENV}` substitution in source URIs; suite-relative CSV/Parquet paths
|
|
22
|
+
- Postgres/MySQL ATTACH stubs (not integration-tested against a live DB)
|
|
23
|
+
|
|
24
|
+
## [0.1.0] - 2026-08-18
|
|
25
|
+
|
|
26
|
+
### Added
|
|
27
|
+
- DuckDB-powered not_null, unique, accepted_values checks
|
|
28
|
+
- YAML suite format and CLI
|
|
@@ -0,0 +1,29 @@
|
|
|
1
|
+
# Contributing
|
|
2
|
+
|
|
3
|
+
## Running tests
|
|
4
|
+
|
|
5
|
+
Prefer Docker Compose:
|
|
6
|
+
|
|
7
|
+
```bash
|
|
8
|
+
docker compose run --rm test
|
|
9
|
+
docker compose run --rm run-example
|
|
10
|
+
```
|
|
11
|
+
|
|
12
|
+
Locally:
|
|
13
|
+
|
|
14
|
+
```bash
|
|
15
|
+
pip install -e ".[dev]"
|
|
16
|
+
pytest tests/ -v
|
|
17
|
+
duckcheck run examples/clean.yaml
|
|
18
|
+
```
|
|
19
|
+
|
|
20
|
+
## Pull requests
|
|
21
|
+
|
|
22
|
+
- Keep example suites intentional (`clean.yaml` passes; `checks.yaml` may fail on purpose)
|
|
23
|
+
- Update README/CHANGELOG for new check types or CLI flags
|
|
24
|
+
- Prefer small PRs with a clear test plan
|
|
25
|
+
|
|
26
|
+
## Commit style
|
|
27
|
+
|
|
28
|
+
- Imperative subject line; mention the user-facing why when relevant
|
|
29
|
+
- Do not add AI co-author trailers (e.g. Co-authored-by: Cursor) to commits.
|
duckcheck-0.4.0/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 duckcheck contributors
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
duckcheck-0.4.0/PKG-INFO
ADDED
|
@@ -0,0 +1,145 @@
|
|
|
1
|
+
Metadata-Version: 2.5
|
|
2
|
+
Name: duckcheck
|
|
3
|
+
Version: 0.4.0
|
|
4
|
+
Summary: Lightweight data quality checks powered by DuckDB
|
|
5
|
+
Project-URL: Homepage, https://github.com/yashshah9/duckcheck
|
|
6
|
+
Project-URL: Repository, https://github.com/yashshah9/duckcheck
|
|
7
|
+
Project-URL: Issues, https://github.com/yashshah9/duckcheck/issues
|
|
8
|
+
Author-email: Yash Shah <yash376351@gmail.com>
|
|
9
|
+
License-Expression: MIT
|
|
10
|
+
License-File: LICENSE
|
|
11
|
+
Requires-Python: >=3.11
|
|
12
|
+
Requires-Dist: click>=8.1
|
|
13
|
+
Requires-Dist: duckdb>=1.0
|
|
14
|
+
Requires-Dist: pydantic>=2.6
|
|
15
|
+
Requires-Dist: pyyaml>=6.0
|
|
16
|
+
Requires-Dist: rich>=13.7
|
|
17
|
+
Requires-Dist: structlog>=24.1
|
|
18
|
+
Provides-Extra: dev
|
|
19
|
+
Requires-Dist: mypy>=1.9; extra == 'dev'
|
|
20
|
+
Requires-Dist: pytest>=8.0; extra == 'dev'
|
|
21
|
+
Requires-Dist: ruff>=0.4; extra == 'dev'
|
|
22
|
+
Requires-Dist: types-pyyaml>=6.0; extra == 'dev'
|
|
23
|
+
Description-Content-Type: text/markdown
|
|
24
|
+
|
|
25
|
+
# duckcheck
|
|
26
|
+
|
|
27
|
+
Lightweight data quality checks powered by **DuckDB** — the anti–Great Expectations for teams who want `pip install`, one YAML file, and one command.
|
|
28
|
+
|
|
29
|
+
[](LICENSE)
|
|
30
|
+
[](https://www.python.org/downloads/)
|
|
31
|
+
[](https://github.com/yashshah9/duckcheck/actions/workflows/ci.yml)
|
|
32
|
+
|
|
33
|
+
> **Status:** v0.4 — CSV/Parquet/SQLite sources, custom SQL with `expect` operators, freshness, baselines, JUnit, and `--format json`.
|
|
34
|
+
|
|
35
|
+
## 60-second try
|
|
36
|
+
|
|
37
|
+
```bash
|
|
38
|
+
docker compose run --rm run-example # duckcheck run examples/clean.yaml
|
|
39
|
+
docker compose run --rm test # pytest
|
|
40
|
+
```
|
|
41
|
+
|
|
42
|
+
## Why this vs alternatives
|
|
43
|
+
|
|
44
|
+
| Approach | Strength | Gap |
|
|
45
|
+
|----------|----------|-----|
|
|
46
|
+
| **duckcheck** | One YAML + DuckDB, local files, CI-friendly | Not a full observability platform |
|
|
47
|
+
| Great Expectations | Rich ecosystem | Heavyweight setup for simple column checks |
|
|
48
|
+
| Soda Core | Familiar check DSL | Cloud-oriented workflow |
|
|
49
|
+
| Ad-hoc SQL in CI | Zero new tools | No standard report / JUnit / baselines |
|
|
50
|
+
|
|
51
|
+
## Problem
|
|
52
|
+
|
|
53
|
+
Data teams need to assert column quality in CI, but Great Expectations is heavyweight and Soda Core funnels to cloud. Ad-hoc SQL checks have no reporting standard.
|
|
54
|
+
|
|
55
|
+
## Key features (v0.4)
|
|
56
|
+
|
|
57
|
+
- YAML check definitions
|
|
58
|
+
- DuckDB scans CSV, Parquet, and SQLite locally — no server
|
|
59
|
+
- Checks: `not_null`, `unique`, `accepted_values`, `custom_sql`, `freshness`, `row_count`, `row_count_delta`
|
|
60
|
+
- `custom_sql` `expect` operators: `0`, `=N`, `>N`, `<N`, `>=N`, `<=N` (default `0`)
|
|
61
|
+
- `--format json` and `--junit` for CI dashboards
|
|
62
|
+
- `${ENV}` in source URIs; `--source-table` for SQL ATTACH
|
|
63
|
+
|
|
64
|
+
## Architecture
|
|
65
|
+
|
|
66
|
+
```
|
|
67
|
+
duckcheck run checks.yaml
|
|
68
|
+
└── SuiteSpec (Pydantic)
|
|
69
|
+
└── DuckDB in-process
|
|
70
|
+
└── source_data view from CSV/Parquet
|
|
71
|
+
```
|
|
72
|
+
|
|
73
|
+
| Component | Technology | Why |
|
|
74
|
+
|-----------|------------|-----|
|
|
75
|
+
| Engine | DuckDB | Single dependency, scans files + SQL databases |
|
|
76
|
+
| CLI | Click + Rich | Simple, good terminal UX |
|
|
77
|
+
| Spec | YAML + Pydantic | Version-controllable checks |
|
|
78
|
+
|
|
79
|
+
## Installation
|
|
80
|
+
|
|
81
|
+
```bash
|
|
82
|
+
pip install duckcheck
|
|
83
|
+
pip install -e ".[dev]"
|
|
84
|
+
```
|
|
85
|
+
|
|
86
|
+
## Usage
|
|
87
|
+
|
|
88
|
+
```bash
|
|
89
|
+
duckcheck health
|
|
90
|
+
duckcheck run examples/clean.yaml
|
|
91
|
+
duckcheck run examples/checks.yaml # fixture with known failures
|
|
92
|
+
duckcheck run examples/clean.yaml --junit /tmp/duckcheck.xml
|
|
93
|
+
duckcheck run examples/clean.yaml --format json
|
|
94
|
+
duckcheck baseline update examples/clean.yaml
|
|
95
|
+
```
|
|
96
|
+
|
|
97
|
+
Example `checks.yaml`:
|
|
98
|
+
|
|
99
|
+
```yaml
|
|
100
|
+
name: sample-suite
|
|
101
|
+
source: examples/sample.csv
|
|
102
|
+
checks:
|
|
103
|
+
- name: id_not_null
|
|
104
|
+
type: not_null
|
|
105
|
+
column: id
|
|
106
|
+
- name: status_values
|
|
107
|
+
type: accepted_values
|
|
108
|
+
column: status
|
|
109
|
+
values: [active, inactive]
|
|
110
|
+
- name: three_active
|
|
111
|
+
type: custom_sql
|
|
112
|
+
sql: "SELECT * FROM source_data WHERE status = 'active'"
|
|
113
|
+
expect: "=3"
|
|
114
|
+
```
|
|
115
|
+
|
|
116
|
+
## Docker
|
|
117
|
+
|
|
118
|
+
```bash
|
|
119
|
+
docker compose run --rm test
|
|
120
|
+
docker compose run --rm run-example
|
|
121
|
+
```
|
|
122
|
+
|
|
123
|
+
## Running tests
|
|
124
|
+
|
|
125
|
+
```bash
|
|
126
|
+
pytest tests/ -v
|
|
127
|
+
```
|
|
128
|
+
|
|
129
|
+
## Roadmap
|
|
130
|
+
|
|
131
|
+
- [x] Freshness + row_count + custom_sql + JUnit
|
|
132
|
+
- [x] Row-count baseline delta store (`duckcheck baseline update`)
|
|
133
|
+
- [x] custom_sql `expect` operators + `--format json`
|
|
134
|
+
- [ ] Live Postgres/MySQL ATTACH integration tests
|
|
135
|
+
- [ ] Airflow/Dagster operators
|
|
136
|
+
|
|
137
|
+
## License
|
|
138
|
+
|
|
139
|
+
MIT
|
|
140
|
+
|
|
141
|
+
## Known limitations (v0.4)
|
|
142
|
+
|
|
143
|
+
- Postgres/MySQL ATTACH is stubbed (`INSTALL/LOAD`) — no live DB in CI yet
|
|
144
|
+
- `examples/checks.yaml` is a failing fixture; `examples/clean.yaml` is the happy path
|
|
145
|
+
- Checks still run against a `source_data` view
|
|
@@ -0,0 +1,121 @@
|
|
|
1
|
+
# duckcheck
|
|
2
|
+
|
|
3
|
+
Lightweight data quality checks powered by **DuckDB** — the anti–Great Expectations for teams who want `pip install`, one YAML file, and one command.
|
|
4
|
+
|
|
5
|
+
[](LICENSE)
|
|
6
|
+
[](https://www.python.org/downloads/)
|
|
7
|
+
[](https://github.com/yashshah9/duckcheck/actions/workflows/ci.yml)
|
|
8
|
+
|
|
9
|
+
> **Status:** v0.4 — CSV/Parquet/SQLite sources, custom SQL with `expect` operators, freshness, baselines, JUnit, and `--format json`.
|
|
10
|
+
|
|
11
|
+
## 60-second try
|
|
12
|
+
|
|
13
|
+
```bash
|
|
14
|
+
docker compose run --rm run-example # duckcheck run examples/clean.yaml
|
|
15
|
+
docker compose run --rm test # pytest
|
|
16
|
+
```
|
|
17
|
+
|
|
18
|
+
## Why this vs alternatives
|
|
19
|
+
|
|
20
|
+
| Approach | Strength | Gap |
|
|
21
|
+
|----------|----------|-----|
|
|
22
|
+
| **duckcheck** | One YAML + DuckDB, local files, CI-friendly | Not a full observability platform |
|
|
23
|
+
| Great Expectations | Rich ecosystem | Heavyweight setup for simple column checks |
|
|
24
|
+
| Soda Core | Familiar check DSL | Cloud-oriented workflow |
|
|
25
|
+
| Ad-hoc SQL in CI | Zero new tools | No standard report / JUnit / baselines |
|
|
26
|
+
|
|
27
|
+
## Problem
|
|
28
|
+
|
|
29
|
+
Data teams need to assert column quality in CI, but Great Expectations is heavyweight and Soda Core funnels to cloud. Ad-hoc SQL checks have no reporting standard.
|
|
30
|
+
|
|
31
|
+
## Key features (v0.4)
|
|
32
|
+
|
|
33
|
+
- YAML check definitions
|
|
34
|
+
- DuckDB scans CSV, Parquet, and SQLite locally — no server
|
|
35
|
+
- Checks: `not_null`, `unique`, `accepted_values`, `custom_sql`, `freshness`, `row_count`, `row_count_delta`
|
|
36
|
+
- `custom_sql` `expect` operators: `0`, `=N`, `>N`, `<N`, `>=N`, `<=N` (default `0`)
|
|
37
|
+
- `--format json` and `--junit` for CI dashboards
|
|
38
|
+
- `${ENV}` in source URIs; `--source-table` for SQL ATTACH
|
|
39
|
+
|
|
40
|
+
## Architecture
|
|
41
|
+
|
|
42
|
+
```
|
|
43
|
+
duckcheck run checks.yaml
|
|
44
|
+
└── SuiteSpec (Pydantic)
|
|
45
|
+
└── DuckDB in-process
|
|
46
|
+
└── source_data view from CSV/Parquet
|
|
47
|
+
```
|
|
48
|
+
|
|
49
|
+
| Component | Technology | Why |
|
|
50
|
+
|-----------|------------|-----|
|
|
51
|
+
| Engine | DuckDB | Single dependency, scans files + SQL databases |
|
|
52
|
+
| CLI | Click + Rich | Simple, good terminal UX |
|
|
53
|
+
| Spec | YAML + Pydantic | Version-controllable checks |
|
|
54
|
+
|
|
55
|
+
## Installation
|
|
56
|
+
|
|
57
|
+
```bash
|
|
58
|
+
pip install duckcheck
|
|
59
|
+
pip install -e ".[dev]"
|
|
60
|
+
```
|
|
61
|
+
|
|
62
|
+
## Usage
|
|
63
|
+
|
|
64
|
+
```bash
|
|
65
|
+
duckcheck health
|
|
66
|
+
duckcheck run examples/clean.yaml
|
|
67
|
+
duckcheck run examples/checks.yaml # fixture with known failures
|
|
68
|
+
duckcheck run examples/clean.yaml --junit /tmp/duckcheck.xml
|
|
69
|
+
duckcheck run examples/clean.yaml --format json
|
|
70
|
+
duckcheck baseline update examples/clean.yaml
|
|
71
|
+
```
|
|
72
|
+
|
|
73
|
+
Example `checks.yaml`:
|
|
74
|
+
|
|
75
|
+
```yaml
|
|
76
|
+
name: sample-suite
|
|
77
|
+
source: examples/sample.csv
|
|
78
|
+
checks:
|
|
79
|
+
- name: id_not_null
|
|
80
|
+
type: not_null
|
|
81
|
+
column: id
|
|
82
|
+
- name: status_values
|
|
83
|
+
type: accepted_values
|
|
84
|
+
column: status
|
|
85
|
+
values: [active, inactive]
|
|
86
|
+
- name: three_active
|
|
87
|
+
type: custom_sql
|
|
88
|
+
sql: "SELECT * FROM source_data WHERE status = 'active'"
|
|
89
|
+
expect: "=3"
|
|
90
|
+
```
|
|
91
|
+
|
|
92
|
+
## Docker
|
|
93
|
+
|
|
94
|
+
```bash
|
|
95
|
+
docker compose run --rm test
|
|
96
|
+
docker compose run --rm run-example
|
|
97
|
+
```
|
|
98
|
+
|
|
99
|
+
## Running tests
|
|
100
|
+
|
|
101
|
+
```bash
|
|
102
|
+
pytest tests/ -v
|
|
103
|
+
```
|
|
104
|
+
|
|
105
|
+
## Roadmap
|
|
106
|
+
|
|
107
|
+
- [x] Freshness + row_count + custom_sql + JUnit
|
|
108
|
+
- [x] Row-count baseline delta store (`duckcheck baseline update`)
|
|
109
|
+
- [x] custom_sql `expect` operators + `--format json`
|
|
110
|
+
- [ ] Live Postgres/MySQL ATTACH integration tests
|
|
111
|
+
- [ ] Airflow/Dagster operators
|
|
112
|
+
|
|
113
|
+
## License
|
|
114
|
+
|
|
115
|
+
MIT
|
|
116
|
+
|
|
117
|
+
## Known limitations (v0.4)
|
|
118
|
+
|
|
119
|
+
- Postgres/MySQL ATTACH is stubbed (`INSTALL/LOAD`) — no live DB in CI yet
|
|
120
|
+
- `examples/checks.yaml` is a failing fixture; `examples/clean.yaml` is the happy path
|
|
121
|
+
- Checks still run against a `source_data` view
|
|
@@ -0,0 +1,14 @@
|
|
|
1
|
+
# Security Policy
|
|
2
|
+
|
|
3
|
+
## Reporting a vulnerability
|
|
4
|
+
|
|
5
|
+
Email **yash376351@gmail.com** with the repo name, a short description, and steps to reproduce. Please do not open a public issue for exploitable findings until we have had a reasonable chance to respond.
|
|
6
|
+
|
|
7
|
+
## Threat model (honest)
|
|
8
|
+
|
|
9
|
+
duckcheck runs SQL checks locally via DuckDB against files or attached databases you point it at.
|
|
10
|
+
|
|
11
|
+
- Custom SQL is executed as written — only load suites and sources you trust.
|
|
12
|
+
- `${ENV}` substitution can pull secrets into URIs; prefer non-secret paths in committed YAML.
|
|
13
|
+
- duckcheck is a **data quality runner**, not an access-control layer for your warehouse.
|
|
14
|
+
- Postgres/MySQL ATTACH helpers are stubs today; do not assume network DB hardening from this tool.
|
|
@@ -0,0 +1,11 @@
|
|
|
1
|
+
services:
|
|
2
|
+
test:
|
|
3
|
+
build: {context: ., dockerfile: docker/Dockerfile}
|
|
4
|
+
volumes: [".:/app"]
|
|
5
|
+
working_dir: /app
|
|
6
|
+
command: ["pytest", "tests/", "-v"]
|
|
7
|
+
run-example:
|
|
8
|
+
build: {context: ., dockerfile: docker/Dockerfile}
|
|
9
|
+
volumes: [".:/app"]
|
|
10
|
+
working_dir: /app
|
|
11
|
+
command: ["duckcheck", "run", "examples/clean.yaml"]
|
|
@@ -0,0 +1,22 @@
|
|
|
1
|
+
name: sample-suite
|
|
2
|
+
source: sample.csv
|
|
3
|
+
checks:
|
|
4
|
+
- name: id_not_null
|
|
5
|
+
type: not_null
|
|
6
|
+
table: source_data
|
|
7
|
+
column: id
|
|
8
|
+
- name: name_not_null
|
|
9
|
+
type: not_null
|
|
10
|
+
table: source_data
|
|
11
|
+
column: name
|
|
12
|
+
- name: status_values
|
|
13
|
+
type: accepted_values
|
|
14
|
+
table: source_data
|
|
15
|
+
column: status
|
|
16
|
+
values: [active, inactive]
|
|
17
|
+
- name: at_least_one_row
|
|
18
|
+
type: row_count
|
|
19
|
+
min_rows: 1
|
|
20
|
+
- name: no_negative_amounts
|
|
21
|
+
type: custom_sql
|
|
22
|
+
sql: SELECT * FROM source_data WHERE amount < 0
|
|
@@ -0,0 +1,19 @@
|
|
|
1
|
+
name: clean-suite
|
|
2
|
+
source: clean.csv
|
|
3
|
+
checks:
|
|
4
|
+
- name: id_not_null
|
|
5
|
+
type: not_null
|
|
6
|
+
column: id
|
|
7
|
+
- name: name_not_null
|
|
8
|
+
type: not_null
|
|
9
|
+
column: name
|
|
10
|
+
- name: status_values
|
|
11
|
+
type: accepted_values
|
|
12
|
+
column: status
|
|
13
|
+
values: [active, inactive]
|
|
14
|
+
- name: at_least_one_row
|
|
15
|
+
type: row_count
|
|
16
|
+
min_rows: 1
|
|
17
|
+
- name: no_negative_amounts
|
|
18
|
+
type: custom_sql
|
|
19
|
+
sql: SELECT * FROM source_data WHERE amount < 0
|
|
@@ -0,0 +1,6 @@
|
|
|
1
|
+
# duckcheck — Planning Documents
|
|
2
|
+
|
|
3
|
+
| Document | Status | Description |
|
|
4
|
+
|----------|--------|-------------|
|
|
5
|
+
| [v0.1-completed-foundation.md](./v0.1-completed-foundation.md) | **Completed** | Initial foundation release |
|
|
6
|
+
| [v0.2-next-phase-roadmap.md](./v0.2-next-phase-roadmap.md) | **In Progress** | v0.2.0 shipped extra checks + JUnit; live ATTACH still open |
|
|
@@ -0,0 +1,200 @@
|
|
|
1
|
+
# v0.1 Completed Foundation Plan — duckcheck
|
|
2
|
+
|
|
3
|
+
**Status:** Completed
|
|
4
|
+
**Version:** 0.1.0
|
|
5
|
+
**Completed:** 2026-08-18
|
|
6
|
+
**Owner:** Yash Shah
|
|
7
|
+
|
|
8
|
+
---
|
|
9
|
+
|
|
10
|
+
## 1. Executive Summary
|
|
11
|
+
|
|
12
|
+
Deliver a **minimal viable open-source foundation** for lightweight data quality checks powered by DuckDB. v0.1 focuses on YAML-defined column checks (`not_null`, `unique`, `accepted_values`) against local CSV/Parquet files — zero server infrastructure, CI-ready from day one via exit codes and Docker.
|
|
13
|
+
|
|
14
|
+
---
|
|
15
|
+
|
|
16
|
+
## 2. Problem Statement
|
|
17
|
+
|
|
18
|
+
- Data teams need column-level quality assertions in CI, but Great Expectations is heavyweight and Soda Core funnels toward cloud
|
|
19
|
+
- Ad-hoc SQL checks have no standard reporting format or reusable spec
|
|
20
|
+
- DuckDB can scan files in-process, but no thin CLI wraps it for check suites
|
|
21
|
+
- Teams want `pip install`, one YAML file, one command — not a platform
|
|
22
|
+
|
|
23
|
+
---
|
|
24
|
+
|
|
25
|
+
## 3. Goals (v0.1)
|
|
26
|
+
|
|
27
|
+
| # | Goal | Status |
|
|
28
|
+
|---|------|--------|
|
|
29
|
+
| G1 | Provide CLI installable Python package | Done |
|
|
30
|
+
| G2 | YAML check suite definitions with Pydantic validation | Done |
|
|
31
|
+
| G3 | Three core check types: `not_null`, `unique`, `accepted_values` | Done |
|
|
32
|
+
| G4 | CSV (and Parquet) source via DuckDB in-process | Done |
|
|
33
|
+
| G5 | Click + Rich CLI with pass/fail exit codes | Done |
|
|
34
|
+
| G6 | Docker-based dev and test environment | Done |
|
|
35
|
+
| G7 | Contributor-ready repo (README, CI, LICENSE) | Done |
|
|
36
|
+
|
|
37
|
+
## 4. Non-Goals (v0.1)
|
|
38
|
+
|
|
39
|
+
- Postgres/MySQL via DuckDB ATTACH
|
|
40
|
+
- Freshness or row-count delta checks
|
|
41
|
+
- JUnit XML or other CI report formats
|
|
42
|
+
- Custom SQL check type
|
|
43
|
+
- Airflow/Dagster operators
|
|
44
|
+
- PyPI publish (deferred to v0.2)
|
|
45
|
+
- Web UI or hosted dashboard
|
|
46
|
+
- Great Expectations compatibility layer
|
|
47
|
+
|
|
48
|
+
---
|
|
49
|
+
|
|
50
|
+
## 5. Architecture Delivered
|
|
51
|
+
|
|
52
|
+
```
|
|
53
|
+
CLI (Click + Rich)
|
|
54
|
+
├── health → version + sanity check
|
|
55
|
+
└── run <suite> → runner.py → DuckDB in-process
|
|
56
|
+
|
|
57
|
+
Models (Pydantic v2)
|
|
58
|
+
├── SuiteSpec → name, source path, checks[]
|
|
59
|
+
└── CheckSpec → name, type, table, column, values[]
|
|
60
|
+
|
|
61
|
+
Engine (DuckDB)
|
|
62
|
+
├── _register_source() → CREATE VIEW source_data
|
|
63
|
+
└── _run_check() → SQL per check type
|
|
64
|
+
|
|
65
|
+
Infrastructure
|
|
66
|
+
├── structlog logging
|
|
67
|
+
└── Docker Compose (test, run-example)
|
|
68
|
+
```
|
|
69
|
+
|
|
70
|
+
### Data Flow
|
|
71
|
+
|
|
72
|
+
```
|
|
73
|
+
checks.yaml
|
|
74
|
+
└── SuiteSpec.model_validate()
|
|
75
|
+
└── duckdb.connect()
|
|
76
|
+
├── read_csv_auto(source) → VIEW source_data
|
|
77
|
+
└── per-check SQL → CheckResult[]
|
|
78
|
+
└── RunReport → Rich table → exit 0|1
|
|
79
|
+
```
|
|
80
|
+
|
|
81
|
+
### Technology Choices
|
|
82
|
+
|
|
83
|
+
| Layer | Choice | Rationale |
|
|
84
|
+
|-------|--------|-----------|
|
|
85
|
+
| Engine | DuckDB ≥1.0 | Single dependency; scans CSV/Parquet locally; future ATTACH for SQL DBs |
|
|
86
|
+
| Language | Python 3.11+ | Data team familiarity, pytest integration |
|
|
87
|
+
| CLI | Click | Mature, composable subcommands |
|
|
88
|
+
| Terminal UX | Rich | Colored pass/fail tables, readable CI output |
|
|
89
|
+
| Validation | Pydantic v2 | Strict YAML schemas, actionable validation errors |
|
|
90
|
+
| Logging | structlog | Structured JSON-capable logs per check |
|
|
91
|
+
| Packaging | hatchling | Modern pyproject.toml builds |
|
|
92
|
+
| Quality | ruff, mypy, pytest | Standard Python OSS toolchain |
|
|
93
|
+
|
|
94
|
+
---
|
|
95
|
+
|
|
96
|
+
## 6. Deliverables Completed
|
|
97
|
+
|
|
98
|
+
### 6.1 Source Code
|
|
99
|
+
|
|
100
|
+
- [x] `src/duckcheck/cli.py` — `health`, `run` commands with Rich table output
|
|
101
|
+
- [x] `src/duckcheck/runner.py` — DuckDB connection, source registration, check execution
|
|
102
|
+
- [x] `src/duckcheck/spec.py` — `SuiteSpec`, `CheckSpec` Pydantic models
|
|
103
|
+
- [x] `src/duckcheck/__init__.py` — package version
|
|
104
|
+
|
|
105
|
+
### 6.2 Check Types Implemented
|
|
106
|
+
|
|
107
|
+
| Check Type | SQL Strategy | Pass Condition |
|
|
108
|
+
|------------|--------------|----------------|
|
|
109
|
+
| `not_null` | `COUNT(*) WHERE col IS NULL` | Count = 0 |
|
|
110
|
+
| `unique` | `COUNT(*) - COUNT(DISTINCT col)` | Duplicates = 0 |
|
|
111
|
+
| `accepted_values` | `COUNT(*) WHERE col NOT IN (...)` | Invalid rows = 0 |
|
|
112
|
+
|
|
113
|
+
### 6.3 Source Formats Supported
|
|
114
|
+
|
|
115
|
+
| Format | DuckDB Function | Status |
|
|
116
|
+
|--------|-----------------|--------|
|
|
117
|
+
| CSV | `read_csv_auto(path)` | Done |
|
|
118
|
+
| Parquet | `read_parquet(path)` | Done |
|
|
119
|
+
| Postgres/MySQL | ATTACH | Deferred v0.2 |
|
|
120
|
+
|
|
121
|
+
### 6.4 Examples & Fixtures
|
|
122
|
+
|
|
123
|
+
- [x] `examples/checks.yaml` — sample suite with intentional failure (`name_not_null`)
|
|
124
|
+
- [x] `examples/sample.csv` — test dataset with null name row
|
|
125
|
+
|
|
126
|
+
### 6.5 Tests
|
|
127
|
+
|
|
128
|
+
- [x] `tests/test_runner.py` — suite execution, failure detection on null column
|
|
129
|
+
- [x] All tests passing in Docker (Python 3.12)
|
|
130
|
+
|
|
131
|
+
### 6.6 DevOps & Documentation
|
|
132
|
+
|
|
133
|
+
- [x] `docker/Dockerfile` — multi-stage dev/runtime
|
|
134
|
+
- [x] `compose.yaml` — `test` and `run-example` services
|
|
135
|
+
- [x] `.github/workflows/ci.yml` — ruff, mypy, pytest
|
|
136
|
+
- [x] `README.md`, `CHANGELOG.md`, `LICENSE`
|
|
137
|
+
- [x] `pyproject.toml` — hatchling build, `[project.scripts]` entry point
|
|
138
|
+
|
|
139
|
+
---
|
|
140
|
+
|
|
141
|
+
## 7. YAML Spec Schema (v0.1)
|
|
142
|
+
|
|
143
|
+
```yaml
|
|
144
|
+
name: sample-suite # required — suite identifier
|
|
145
|
+
source: examples/sample.csv # required — path to CSV or Parquet
|
|
146
|
+
checks:
|
|
147
|
+
- name: id_not_null # required — unique check name
|
|
148
|
+
type: not_null # required — not_null | unique | accepted_values
|
|
149
|
+
table: source_data # required — view name (always source_data in v0.1)
|
|
150
|
+
column: id # required for column checks
|
|
151
|
+
- name: status_values
|
|
152
|
+
type: accepted_values
|
|
153
|
+
table: source_data
|
|
154
|
+
column: status
|
|
155
|
+
values: [active, inactive] # required for accepted_values
|
|
156
|
+
```
|
|
157
|
+
|
|
158
|
+
---
|
|
159
|
+
|
|
160
|
+
## 8. Acceptance Criteria (Met)
|
|
161
|
+
|
|
162
|
+
| # | Criterion | Verification |
|
|
163
|
+
|---|-----------|--------------|
|
|
164
|
+
| AC1 | `duckcheck run examples/checks.yaml` exits non-zero on failure | Manual + Docker `run-example` |
|
|
165
|
+
| AC2 | Rich table shows PASS/FAIL per check with message | Manual |
|
|
166
|
+
| AC3 | `docker compose run --rm test` passes all tests | CI-ready |
|
|
167
|
+
| AC4 | Package installable via `pip install -e .` on Python 3.11+ | pyproject.toml |
|
|
168
|
+
| AC5 | CSV source registered as `source_data` view | runner.py unit test |
|
|
169
|
+
| AC6 | Unknown check type returns FAIL with descriptive message | runner.py |
|
|
170
|
+
| AC7 | No secrets or credentials in repository | .gitignore, no .env committed |
|
|
171
|
+
|
|
172
|
+
---
|
|
173
|
+
|
|
174
|
+
## 9. Known Limitations (v0.1)
|
|
175
|
+
|
|
176
|
+
- Single source per suite — no multi-table joins
|
|
177
|
+
- `table` field in YAML is informational; all checks query `source_data` view
|
|
178
|
+
- No parameterized connection strings for remote databases
|
|
179
|
+
- No JUnit XML — CI must parse stdout or rely on exit code only
|
|
180
|
+
- Column names interpolated into SQL without quoting (safe for simple identifiers only)
|
|
181
|
+
- Parquet supported in code but not covered by example fixtures
|
|
182
|
+
|
|
183
|
+
---
|
|
184
|
+
|
|
185
|
+
## 10. Metrics at Completion
|
|
186
|
+
|
|
187
|
+
| Metric | Value |
|
|
188
|
+
|--------|-------|
|
|
189
|
+
| Source files | 4 Python modules |
|
|
190
|
+
| Test cases | 1 (integration-style) |
|
|
191
|
+
| CLI commands | 2 (`health`, `run`) |
|
|
192
|
+
| Check types | 3 |
|
|
193
|
+
| Source formats | 2 (CSV, Parquet) |
|
|
194
|
+
| Docker services | 2 (`test`, `run-example`) |
|
|
195
|
+
|
|
196
|
+
---
|
|
197
|
+
|
|
198
|
+
## 11. Sign-Off
|
|
199
|
+
|
|
200
|
+
v0.1 foundation is **complete and shippable** as an alpha OSS project. The DuckDB + YAML + Rich CLI stack provides immediate value for local CSV/Parquet quality gates without server infrastructure. Database ATTACH, freshness checks, JUnit output, and orchestrator integration are deferred to v0.2 per roadmap.
|