cellproof 0.2.0__tar.gz → 0.2.2__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {cellproof-0.2.0 → cellproof-0.2.2}/AGENTS.md +1 -1
- cellproof-0.2.2/CHANGELOG.md +50 -0
- cellproof-0.2.2/INFO.md +153 -0
- {cellproof-0.2.0 → cellproof-0.2.2}/PKG-INFO +21 -11
- {cellproof-0.2.0 → cellproof-0.2.2}/PROGRESS.md +18 -6
- {cellproof-0.2.0 → cellproof-0.2.2}/README.md +19 -9
- cellproof-0.2.2/docs/JOURNEY.md +266 -0
- {cellproof-0.2.0 → cellproof-0.2.2}/pyproject.toml +2 -2
- cellproof-0.2.2/src/cellproof/__init__.py +1 -0
- {cellproof-0.2.0 → cellproof-0.2.2}/src/cellproof/compiler.py +66 -3
- {cellproof-0.2.0 → cellproof-0.2.2}/src/cellproof/expr.py +40 -2
- {cellproof-0.2.0 → cellproof-0.2.2}/src/cellproof/fn/aggregate.py +17 -1
- {cellproof-0.2.0 → cellproof-0.2.2}/src/cellproof/fn/dates.py +60 -6
- {cellproof-0.2.0 → cellproof-0.2.2}/src/cellproof/fn/logic.py +7 -8
- {cellproof-0.2.0 → cellproof-0.2.2}/src/cellproof/fn/lookup.py +29 -1
- {cellproof-0.2.0 → cellproof-0.2.2}/src/cellproof/fn/maths.py +4 -1
- {cellproof-0.2.0 → cellproof-0.2.2}/src/cellproof/fn/text.py +10 -6
- {cellproof-0.2.0 → cellproof-0.2.2}/src/cellproof/functions.py +38 -9
- {cellproof-0.2.0 → cellproof-0.2.2}/src/cellproof/model.py +2 -0
- {cellproof-0.2.0 → cellproof-0.2.2}/src/cellproof/parser.py +77 -6
- {cellproof-0.2.0 → cellproof-0.2.2}/src/cellproof/r1c1.py +33 -3
- {cellproof-0.2.0 → cellproof-0.2.2}/src/cellproof/reader.py +64 -6
- {cellproof-0.2.0 → cellproof-0.2.2}/src/cellproof/refs.py +3 -1
- {cellproof-0.2.0 → cellproof-0.2.2}/src/cellproof/verifier.py +4 -0
- cellproof-0.2.2/tests/corpus.py +50 -0
- cellproof-0.2.2/tests/test_arrays.py +45 -0
- cellproof-0.2.2/tests/test_corpus_findings.py +142 -0
- {cellproof-0.2.0 → cellproof-0.2.2}/tests/test_dates.py +26 -0
- {cellproof-0.2.0 → cellproof-0.2.2}/tests/test_emitter.py +11 -0
- cellproof-0.2.2/tests/test_implicit.py +77 -0
- {cellproof-0.2.0 → cellproof-0.2.2}/tests/test_lexed.py +1 -0
- {cellproof-0.2.0 → cellproof-0.2.2}/tests/test_lookup.py +29 -0
- {cellproof-0.2.0 → cellproof-0.2.2}/tests/test_parser.py +1 -1
- {cellproof-0.2.0 → cellproof-0.2.2}/tests/test_reader.py +17 -0
- {cellproof-0.2.0 → cellproof-0.2.2}/tests/test_region.py +5 -0
- {cellproof-0.2.0 → cellproof-0.2.2}/tests/test_text.py +2 -1
- {cellproof-0.2.0 → cellproof-0.2.2}/tests/test_types.py +1 -1
- cellproof-0.2.0/CHANGELOG.md +0 -17
- cellproof-0.2.0/src/cellproof/__init__.py +0 -1
- {cellproof-0.2.0 → cellproof-0.2.2}/.github/workflows/ci.yml +0 -0
- {cellproof-0.2.0 → cellproof-0.2.2}/.github/workflows/publish.yml +0 -0
- {cellproof-0.2.0 → cellproof-0.2.2}/.gitignore +0 -0
- {cellproof-0.2.0 → cellproof-0.2.2}/DESIGN.md +0 -0
- {cellproof-0.2.0 → cellproof-0.2.2}/LICENSE +0 -0
- {cellproof-0.2.0 → cellproof-0.2.2}/PLAN-2a.md +0 -0
- {cellproof-0.2.0 → cellproof-0.2.2}/PLAN-2b.md +0 -0
- {cellproof-0.2.0 → cellproof-0.2.2}/PLAN.md +0 -0
- {cellproof-0.2.0 → cellproof-0.2.2}/SECURITY.md +0 -0
- {cellproof-0.2.0 → cellproof-0.2.2}/docs/superpowers/plans/2026-10-14-plan-2c-speed.md +0 -0
- {cellproof-0.2.0 → cellproof-0.2.2}/docs/superpowers/plans/2026-10-14-plan-2d-table-mode.md +0 -0
- {cellproof-0.2.0 → cellproof-0.2.2}/docs/superpowers/specs/2026-10-14-plan-2c-speed-design.md +0 -0
- {cellproof-0.2.0 → cellproof-0.2.2}/docs/superpowers/specs/2026-10-14-plan-2d-table-mode-design.md +0 -0
- {cellproof-0.2.0 → cellproof-0.2.2}/docs/superpowers/specs/2026-10-18-running-totals-design.md +0 -0
- {cellproof-0.2.0 → cellproof-0.2.2}/docs/superpowers/specs/2026-10-19-lookups-row-wise-design.md +0 -0
- {cellproof-0.2.0 → cellproof-0.2.2}/examples/commission.xlsx +0 -0
- {cellproof-0.2.0 → cellproof-0.2.2}/examples/errors.xlsx +0 -0
- {cellproof-0.2.0 → cellproof-0.2.2}/examples/functions.xlsx +0 -0
- {cellproof-0.2.0 → cellproof-0.2.2}/examples/functions_unsaved.xlsx +0 -0
- {cellproof-0.2.0 → cellproof-0.2.2}/examples/make_commission.py +0 -0
- {cellproof-0.2.0 → cellproof-0.2.2}/examples/make_errors.py +0 -0
- {cellproof-0.2.0 → cellproof-0.2.2}/examples/make_functions.py +0 -0
- {cellproof-0.2.0 → cellproof-0.2.2}/src/cellproof/__main__.py +0 -0
- {cellproof-0.2.0 → cellproof-0.2.2}/src/cellproof/cli.py +0 -0
- {cellproof-0.2.0 → cellproof-0.2.2}/src/cellproof/compute.py +0 -0
- {cellproof-0.2.0 → cellproof-0.2.2}/src/cellproof/emitter.py +0 -0
- {cellproof-0.2.0 → cellproof-0.2.2}/src/cellproof/fenwick.py +0 -0
- {cellproof-0.2.0 → cellproof-0.2.2}/src/cellproof/fn/__init__.py +0 -0
- {cellproof-0.2.0 → cellproof-0.2.2}/src/cellproof/fn/criteria.py +0 -0
- {cellproof-0.2.0 → cellproof-0.2.2}/src/cellproof/graph.py +0 -0
- {cellproof-0.2.0 → cellproof-0.2.2}/src/cellproof/rangeindex.py +0 -0
- {cellproof-0.2.0 → cellproof-0.2.2}/src/cellproof/region.py +0 -0
- {cellproof-0.2.0 → cellproof-0.2.2}/src/cellproof/regions.py +0 -0
- {cellproof-0.2.0 → cellproof-0.2.2}/src/cellproof/report.py +0 -0
- {cellproof-0.2.0 → cellproof-0.2.2}/tests/bench_shapes.py +0 -0
- {cellproof-0.2.0 → cellproof-0.2.2}/tests/test_aggregate.py +0 -0
- {cellproof-0.2.0 → cellproof-0.2.2}/tests/test_batches.py +0 -0
- {cellproof-0.2.0 → cellproof-0.2.2}/tests/test_cli.py +0 -0
- {cellproof-0.2.0 → cellproof-0.2.2}/tests/test_compiler.py +0 -0
- {cellproof-0.2.0 → cellproof-0.2.2}/tests/test_compute.py +0 -0
- {cellproof-0.2.0 → cellproof-0.2.2}/tests/test_criteria.py +0 -0
- {cellproof-0.2.0 → cellproof-0.2.2}/tests/test_errors.py +0 -0
- {cellproof-0.2.0 → cellproof-0.2.2}/tests/test_excel_findings.py +0 -0
- {cellproof-0.2.0 → cellproof-0.2.2}/tests/test_excel_saved.py +0 -0
- {cellproof-0.2.0 → cellproof-0.2.2}/tests/test_fenwick.py +0 -0
- {cellproof-0.2.0 → cellproof-0.2.2}/tests/test_functions.py +0 -0
- {cellproof-0.2.0 → cellproof-0.2.2}/tests/test_graph.py +0 -0
- {cellproof-0.2.0 → cellproof-0.2.2}/tests/test_index.py +0 -0
- {cellproof-0.2.0 → cellproof-0.2.2}/tests/test_limits.py +0 -0
- {cellproof-0.2.0 → cellproof-0.2.2}/tests/test_logic.py +0 -0
- {cellproof-0.2.0 → cellproof-0.2.2}/tests/test_lookups.py +0 -0
- {cellproof-0.2.0 → cellproof-0.2.2}/tests/test_maths.py +0 -0
- {cellproof-0.2.0 → cellproof-0.2.2}/tests/test_perf.py +0 -0
- {cellproof-0.2.0 → cellproof-0.2.2}/tests/test_r1c1.py +0 -0
- {cellproof-0.2.0 → cellproof-0.2.2}/tests/test_rangeindex.py +0 -0
- {cellproof-0.2.0 → cellproof-0.2.2}/tests/test_refs.py +0 -0
- {cellproof-0.2.0 → cellproof-0.2.2}/tests/test_regions.py +0 -0
- {cellproof-0.2.0 → cellproof-0.2.2}/tests/test_regions_differential.py +0 -0
- {cellproof-0.2.0 → cellproof-0.2.2}/tests/test_regions_split.py +0 -0
- {cellproof-0.2.0 → cellproof-0.2.2}/tests/test_report.py +0 -0
- {cellproof-0.2.0 → cellproof-0.2.2}/tests/test_review_regressions.py +0 -0
- {cellproof-0.2.0 → cellproof-0.2.2}/tests/test_running.py +0 -0
- {cellproof-0.2.0 → cellproof-0.2.2}/tests/test_shapes.py +0 -0
- {cellproof-0.2.0 → cellproof-0.2.2}/tests/test_verifier.py +0 -0
- {cellproof-0.2.0 → cellproof-0.2.2}/tests/xlsxkit.py +0 -0
|
@@ -28,7 +28,7 @@ CELLPROOF_PERF=1 python -m pytest tests/test_perf.py -s # 200k-formula benchmar
|
|
|
28
28
|
| `expr.py` | `Expr` (SQL value + SQL error code), Excel coercions |
|
|
29
29
|
| `compiler.py` | cell mode: formula -> SQL in batches |
|
|
30
30
|
| `region.py` | a region -> one set-based statement (window functions for running totals) |
|
|
31
|
-
| `functions.py`, `fn/` | the
|
|
31
|
+
| `functions.py`, `fn/` | the 96 Excel functions |
|
|
32
32
|
| `verifier.py` | runs everything on DuckDB and compares with Excel's saved values |
|
|
33
33
|
| `emitter.py` | writes the dbt project; `--dbt` runs `dbt build` |
|
|
34
34
|
| `report.py`, `cli.py`, `compute.py` | REPORT.md/report.json, the CLI, the optional `formulas` answer key |
|
|
@@ -0,0 +1,50 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
## 0.2.2 (2026-10-27)
|
|
4
|
+
|
|
5
|
+
- **LOOKUP** (vector and array form). Like the other approximate lookups, UNRESOLVED on data out of order.
|
|
6
|
+
- **YEARFRAC**, all five bases, the basis written in the formula or computed. The 1,992 YEARFRAC cells of pycel's
|
|
7
|
+
test workbook all PASS; a date before 1 March 1900 (Excel's 29 February 1900) is UNRESOLVED.
|
|
8
|
+
- **Array constants** (`{0,50,100}`, `{1,"a";2,"b"}`) wherever a range is taken: `LOOKUP(x,{0,50,100},{"F","C","A"})`,
|
|
9
|
+
`MATCH(x,{...},0)`, `SUM({...})`. Used as one value (`A1*{1,2}`) they stay UNRESOLVED.
|
|
10
|
+
|
|
11
|
+
## 0.2.1 (2026-10-20)
|
|
12
|
+
|
|
13
|
+
Found by running all 85 public test workbooks of pycel, formulas and xlcalculator (`python tests/corpus.py DIR`):
|
|
14
|
+
FAIL 229 -> 10 (all 10 in files LibreOffice saved, which the report now says), no crashes.
|
|
15
|
+
|
|
16
|
+
- **Reading:** a defined name that is a formula of names (`=h+i`) no longer crashes the load; a date-formatted number
|
|
17
|
+
keeps its serial (Excel's 29 February 1900 and date-formatted negatives were lost); text escaped by the file format
|
|
18
|
+
(`_x000D_`, written for a carriage return) is decoded as Excel does.
|
|
19
|
+
- **AND/OR/XOR** skip text and empty cells in references, and typed text other than TRUE/FALSE, like Excel.
|
|
20
|
+
- **An empty or omitted result is 0:** `IF(c,,"x")`, `IFERROR(A1,"x")` with A1 empty.
|
|
21
|
+
- **EDATE/EOMONTH/MROUND/VALUE** refuse TRUE/FALSE (`#VALUE!`); `EOMONTH(0,0)` is 31.
|
|
22
|
+
- **SUMPRODUCT** of ranges of different sizes returns an error inside them before `#VALUE!`.
|
|
23
|
+
- **PROPER** accepts another cell's value (DuckDB refused the SQL) and capitalizes accented letters.
|
|
24
|
+
- **Locale-dependent text** (`"1,000"`, `"1/2/2020"`) is UNRESOLVED instead of a wrong `#VALUE!`.
|
|
25
|
+
- **Report:** warns when the file was not last saved by Excel; CHAR above 127 in a Mac-calculated file is UNRESOLVED.
|
|
26
|
+
|
|
27
|
+
From a user's practice workbook (3 PASS / 47 UNRESOLVED -> 48 PASS / 2 UNRESOLVED):
|
|
28
|
+
|
|
29
|
+
- **Implicit intersection:** a one-column or one-row range where one value is expected takes the formula's own row or
|
|
30
|
+
column (`=IF(E$3:E$41="F",B$3:B$41)`); outside it is `#VALUE!`. A range IF passes into another function is
|
|
31
|
+
UNRESOLVED (Excel gives that function the whole range).
|
|
32
|
+
- **`INDIRECT("Sales")`** of fixed text is the reference it names (dependencies included).
|
|
33
|
+
- **MODE / MODE.SNGL.**
|
|
34
|
+
- **A bare word is a name, never a column:** `=i` without a name `i` was read as column I.
|
|
35
|
+
|
|
36
|
+
## 0.2.0 (2026-10-19)
|
|
37
|
+
|
|
38
|
+
- **Renamed** from `unexcel` to `cellproof` (the old name clashed with an R package): `pip install cellproof`, command
|
|
39
|
+
`cellproof convert`, import `cellproof`, `CELLPROOF_PERF` for the benchmark, dbt project and database `cellproof`.
|
|
40
|
+
- **Lookups row-wise:** VLOOKUP, HLOOKUP, MATCH, INDEX and XLOOKUP pasted down a column compile into one statement.
|
|
41
|
+
1,000 VLOOKUPs: verify 5.7 s -> 0.1 s.
|
|
42
|
+
- **Criteria row-wise:** COUNTIFS (and SUMIF/AVERAGEIF(S) over a range holding an error) with a criterion from the row
|
|
43
|
+
no longer fall back to cell mode. 1,000 COUNTIFS: verify 29 s -> 0.1 s.
|
|
44
|
+
- **Faster compile:** a pasted-down formula is tokenized once per run. 200k-formula benchmark 31 s -> 26 s; 50k
|
|
45
|
+
VLOOKUPs 8.9 s -> 4.6 s.
|
|
46
|
+
- A lookup outside a region is about 20% slower than in 0.1.0 (one SQL form for both modes).
|
|
47
|
+
|
|
48
|
+
## 0.1.0 (2026-10-18)
|
|
49
|
+
|
|
50
|
+
First release, as `unexcel`.
|
cellproof-0.2.2/INFO.md
ADDED
|
@@ -0,0 +1,153 @@
|
|
|
1
|
+
# INFO: handoff for the next coding agent
|
|
2
|
+
|
|
3
|
+
You are taking over development of **cellproof**. Read this file first, then `AGENTS.md` (rules), `README.md` (what the
|
|
4
|
+
tool does and its limits) and `PROGRESS.md` (state and next steps). This file is the short version of all of them plus
|
|
5
|
+
the things that are not written down elsewhere.
|
|
6
|
+
|
|
7
|
+
Last updated: 2026-10-27. Current release: 0.2.2 (PyPI: `cellproof`). Repo: github.com/Nikhil414/cellproof (public).
|
|
8
|
+
|
|
9
|
+
## What we are building
|
|
10
|
+
|
|
11
|
+
cellproof turns an Excel workbook (`.xlsx`/`.xlsm`) into SQL and **proves every number matches**.
|
|
12
|
+
|
|
13
|
+
1. Read the formulas and the values Excel saved in the file (Excel's saved values are a free answer key).
|
|
14
|
+
2. Translate every formula to SQL (DuckDB dialect), with Excel's exact rules: type coercions, error codes, error order.
|
|
15
|
+
3. Run the SQL on DuckDB and compare each result with Excel's saved value.
|
|
16
|
+
4. Emit a dbt project that recomputes the workbook, plus `REPORT.md` / `report.json`.
|
|
17
|
+
|
|
18
|
+
Every formula cell gets one verdict:
|
|
19
|
+
|
|
20
|
+
- **PASS**: SQL result equals Excel's saved value.
|
|
21
|
+
- **FAIL**: SQL result differs; the report shows both values.
|
|
22
|
+
- **UNRESOLVED**: cellproof cannot translate it exactly; the report says why (the text of an `Unsupported` exception).
|
|
23
|
+
|
|
24
|
+
## End goal
|
|
25
|
+
|
|
26
|
+
A finance or analytics person who has business logic trapped in a spreadsheet can move it to a database and **show
|
|
27
|
+
evidence** that nothing changed. The product is trust, not coverage: an honest UNRESOLVED beats a guessed PASS.
|
|
28
|
+
|
|
29
|
+
Longer term:
|
|
30
|
+
|
|
31
|
+
- Cover the Excel features real business workbooks use (array formulas are the biggest missing piece).
|
|
32
|
+
- Stay fast on big workbooks (200k formulas verifies in about 26 s today).
|
|
33
|
+
- Plan 3 (not started): an AI retry loop for cells that come out UNRESOLVED or FAIL. It may see **formula structure
|
|
34
|
+
only, never cell values**.
|
|
35
|
+
- Other warehouses besides DuckDB for the dbt output (not started).
|
|
36
|
+
|
|
37
|
+
## Non-negotiable rules (from AGENTS.md)
|
|
38
|
+
|
|
39
|
+
- **Never guess a value.** Anything not translated exactly raises `Unsupported("<reason a user understands>")`.
|
|
40
|
+
- **A false PASS is the worst bug.** When unsure, produce an honest FAIL or UNRESOLVED.
|
|
41
|
+
- **No DuckDB exception may escape from a supported function.** Guard every domain with `CASE`.
|
|
42
|
+
- **Regions must equal cell mode.** `compile_book(book, regions=False)` is the reference; region tests compare to it.
|
|
43
|
+
- **Runtime dependencies stay `openpyxl` and `duckdb`.** Anything else goes in an optional extra.
|
|
44
|
+
- **Cell values never leave the machine.** No network calls with workbook data.
|
|
45
|
+
- **Tests first.** Console output is ASCII-only. Measure before claiming speed.
|
|
46
|
+
- **Real Excel is the ground truth.** The `formulas` library is only a labelled fallback answer key.
|
|
47
|
+
- Commits made by an agent end with a `Co-Authored-By:` trailer naming the agent.
|
|
48
|
+
|
|
49
|
+
## Where we are (2026-10-27)
|
|
50
|
+
|
|
51
|
+
Shipped and on `main` / PyPI 0.2.2:
|
|
52
|
+
|
|
53
|
+
- Full pipeline: reader, parser, dependency graph, cell-mode compiler, region (table) mode, verifier, dbt emitter, report, CLI.
|
|
54
|
+
- 96 Excel functions (list in README). Latest additions: LOOKUP, YEARFRAC (all 5 bases, dynamic basis), array constants
|
|
55
|
+
such as `{0,50,100}` wherever a range is taken.
|
|
56
|
+
- Region mode: a column repeating one formula compiles to one set-based statement; running totals become window
|
|
57
|
+
functions; VLOOKUP/HLOOKUP/MATCH/INDEX/XLOOKUP and the *IF(S) family pasted down compile to joins.
|
|
58
|
+
- Speed: 200k formulas in 26 s; 50k-row VLOOKUP in 4.6 s.
|
|
59
|
+
- Tests: about 430 pass, 8 skip (dbt tests skip when `dbt` is not on PATH). Whole suite takes 4 to 8 minutes.
|
|
60
|
+
- Validation: 3 workbooks saved by real Excel all PASS. 85 public test workbooks (pycel, formulas, xlcalculator):
|
|
61
|
+
6,874 PASS, 10 FAIL (all in LibreOffice-saved files, flagged in their reports), 16,764 UNRESOLVED, no crashes.
|
|
62
|
+
Run it with `python tests/corpus.py DIR`.
|
|
63
|
+
- CI: Windows, macOS and Linux x Python 3.10 to 3.13. Release = GitHub release, which triggers the PyPI publish workflow.
|
|
64
|
+
|
|
65
|
+
## What to do next (in order)
|
|
66
|
+
|
|
67
|
+
1. **Array formulas and array evaluation.** `SUM(A1:A10*B1:B10)`, `MAX(IF(c, A1:A9, 0))`, `A1*{1,2}`. Today these are
|
|
68
|
+
UNRESOLVED. This is the largest remaining source of UNRESOLVED cells in business workbooks. It needs element-wise
|
|
69
|
+
evaluation of ranges, which is a new design piece: write a short spec in `docs/superpowers/specs/` first, then a plan
|
|
70
|
+
in `docs/superpowers/plans/`, then build test-first. Existing building blocks you can reuse:
|
|
71
|
+
- Array constants are parsed into hidden-sheet `RefNode`s; the values live in an `ARRAYS` dict keyed by content hash.
|
|
72
|
+
The hidden sheet name is `#array-<hash>` (see gotchas for why).
|
|
73
|
+
- Regions already compile column-wise statements; element-wise range math may fit there.
|
|
74
|
+
2. **Test on more real analyst workbooks.** One user workbook found 4 gaps the 85 public files did not. Every real
|
|
75
|
+
workbook is worth more than another synthetic test. Turn each finding into a test in
|
|
76
|
+
`tests/test_corpus_findings.py`.
|
|
77
|
+
3. **Read time.** openpyxl loads the workbook twice (8.3 s of the 26 s). Also the per-cell dependency loop (11.3 s compile).
|
|
78
|
+
4. **Plan 3**: AI retry loop (structure only, never values).
|
|
79
|
+
|
|
80
|
+
Known smaller gaps are listed in `PROGRESS.md` under "Known gaps".
|
|
81
|
+
|
|
82
|
+
## Code map (`src/cellproof/`)
|
|
83
|
+
|
|
84
|
+
| Module | Job |
|
|
85
|
+
|---|---|
|
|
86
|
+
| `model.py` | `Cell`, `Book`, `Verdict`, `Unsupported` (its message becomes the UNRESOLVED reason) |
|
|
87
|
+
| `reader.py` | loads the workbook twice with openpyxl (formulas, saved values); refuses zip bombs |
|
|
88
|
+
| `parser.py`, `refs.py` | formula text to AST with Excel precedence; reference text to `Area` |
|
|
89
|
+
| `r1c1.py` | how a reference behaves when a formula is pasted down |
|
|
90
|
+
| `regions.py` | finds regions, running totals, splits on cycles |
|
|
91
|
+
| `graph.py`, `fenwick.py`, `rangeindex.py` | dependency order without expanding ranges; cycle detection |
|
|
92
|
+
| `expr.py` | `Expr` (SQL value + SQL error code), Excel coercions, sentinels such as `#1900` |
|
|
93
|
+
| `compiler.py` | cell mode: formula to SQL in batches |
|
|
94
|
+
| `region.py` | a region to one set-based statement |
|
|
95
|
+
| `functions.py`, `fn/` | the Excel functions (`fn/lookup.py`, `fn/dates.py`, ...) |
|
|
96
|
+
| `verifier.py` | runs everything on DuckDB and compares with Excel's saved values |
|
|
97
|
+
| `emitter.py` | writes the dbt project; `--dbt` runs `dbt build` |
|
|
98
|
+
| `report.py`, `cli.py`, `compute.py` | REPORT.md/report.json, the CLI, the optional `formulas` answer key |
|
|
99
|
+
|
|
100
|
+
## How to work here
|
|
101
|
+
|
|
102
|
+
Setup (Mac/Linux):
|
|
103
|
+
|
|
104
|
+
```bash
|
|
105
|
+
python -m venv .venv && . .venv/bin/activate
|
|
106
|
+
pip install -e ".[dev,dbt,compute]"
|
|
107
|
+
python -m pytest tests -q # dbt must be on PATH (venv active) or the dbt tests skip
|
|
108
|
+
```
|
|
109
|
+
|
|
110
|
+
Workflow that has worked for every piece so far:
|
|
111
|
+
|
|
112
|
+
1. Write the failing tests first (TDD). Add a region-mode test next to every new function; it must equal cell mode.
|
|
113
|
+
2. Implement the smallest thing that is exactly right. If it cannot be exact, raise `Unsupported` with a plain reason.
|
|
114
|
+
3. Run the whole suite. After a change that can affect results, re-run the corpus and compare PASS/FAIL/UNRESOLVED
|
|
115
|
+
counts. FAIL must not go up.
|
|
116
|
+
4. Update `README.md` (function list and limitations), `CHANGELOG.md` (under "Unreleased") and `PROGRESS.md`.
|
|
117
|
+
5. When Excel's behaviour is uncertain, **do not guess from memory**. Remove the unverified test case, or find an
|
|
118
|
+
Excel-saved workbook that proves it. (Example: a YEARFRAC test with an unverified expected value was dropped.)
|
|
119
|
+
|
|
120
|
+
## Gotchas learned the hard way
|
|
121
|
+
|
|
122
|
+
- **Braces in sheet names break things.** Python `.format()` and dbt's Jinja both treat `{...}` as placeholders. The
|
|
123
|
+
array sheet used to be `{cellproof-array-hash}` and failed in region mode and in dbt; it is now `#array-<hash>`. Never
|
|
124
|
+
put `{` or `}` in generated names. Detect array sheets by membership in the `ARRAYS` dict, not by a string prefix.
|
|
125
|
+
- **Error order matters.** Excel reports errors left to right, and errors typed into a formula come before errors inside
|
|
126
|
+
referenced cells. YEARFRAC needed the basis error checked before date errors (52 corpus FAILs until fixed).
|
|
127
|
+
- **Parser tests that expect "unparseable" input go stale** when the parser learns new syntax (`{1,2}` was one). Pick a
|
|
128
|
+
form that is still invalid (`{1,A1}`).
|
|
129
|
+
- **Excel facts verified against real Excel** are in `PROGRESS.md` ("What Excel taught us") with tests in
|
|
130
|
+
`tests/test_excel_findings.py`. Read them before touching COUNTIF, DAYS360, TEXT, MOD, POWER or date code.
|
|
131
|
+
- **Approximate lookups on unsorted data** (LOOKUP, MATCH/VLOOKUP with type 1/-1) stay UNRESOLVED. Excel's answer there
|
|
132
|
+
depends on its binary search.
|
|
133
|
+
- **Text to number** only handles plain numbers. Locale-dependent text (`"1,000"`, `"$5"`, `"50%"`) is UNRESOLVED.
|
|
134
|
+
- Console output must stay ASCII (the tool is also used on Windows consoles).
|
|
135
|
+
- No linter is installed in the dev environment (ruff and pyflakes are missing). Keep the existing style by hand.
|
|
136
|
+
|
|
137
|
+
## Repo and release facts
|
|
138
|
+
|
|
139
|
+
- Public repo `Nikhil414/cellproof` (remote `origin`), single-commit-history start. **Never push the old history
|
|
140
|
+
publicly.** A private backup `Nikhil414/unexcel-dev` exists (remote `dev`, branch `public-main` mirrors `main`).
|
|
141
|
+
The project was called `unexcel` until 0.2.0; older plans and specs still use that name.
|
|
142
|
+
- Release: bump `version` in `pyproject.toml` and `src/cellproof/__init__.py`, date the CHANGELOG, run the tests,
|
|
143
|
+
commit, push `main`, then `gh release create vX.Y.Z`. The release triggers the PyPI publish workflow. Confirm CI is
|
|
144
|
+
green first.
|
|
145
|
+
- Only push, tag or release when the owner asks.
|
|
146
|
+
- The owner prefers decisions to questions: decide, build, test, security-check, and explain what you did.
|
|
147
|
+
|
|
148
|
+
## Safety checklist before every commit
|
|
149
|
+
|
|
150
|
+
- No workbook data, secrets or tokens in the repo, tests or logs.
|
|
151
|
+
- No new runtime dependency.
|
|
152
|
+
- No network call that carries workbook data.
|
|
153
|
+
- A false PASS is possible? Make it an UNRESOLVED instead.
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.5
|
|
2
2
|
Name: cellproof
|
|
3
|
-
Version: 0.2.
|
|
3
|
+
Version: 0.2.2
|
|
4
4
|
Summary: Turn an Excel workbook into SQL (a dbt project on DuckDB) and prove every number matches.
|
|
5
5
|
Project-URL: Homepage, https://github.com/Nikhil414/cellproof
|
|
6
6
|
Project-URL: Issues, https://github.com/Nikhil414/cellproof/issues
|
|
@@ -21,7 +21,7 @@ Classifier: Programming Language :: Python :: 3.13
|
|
|
21
21
|
Classifier: Topic :: Database
|
|
22
22
|
Classifier: Topic :: Office/Business :: Financial :: Spreadsheet
|
|
23
23
|
Requires-Python: >=3.10
|
|
24
|
-
Requires-Dist: duckdb>=1.
|
|
24
|
+
Requires-Dist: duckdb>=1.3
|
|
25
25
|
Requires-Dist: openpyxl>=3.1
|
|
26
26
|
Provides-Extra: compute
|
|
27
27
|
Requires-Dist: formulas<1.4,>=1.3.4; extra == 'compute'
|
|
@@ -110,12 +110,12 @@ reader -> regions -> dependency graph -> compiler -> verifier -> dbt emitter ->
|
|
|
110
110
|
code.
|
|
111
111
|
6. **Emit** the dbt project and the report.
|
|
112
112
|
|
|
113
|
-
|
|
113
|
+
96 Excel functions are supported: ABS, AND, AVERAGE, AVERAGEIF(S), CEILING, CHAR, CHOOSE, CLEAN, CONCAT(ENATE),
|
|
114
114
|
COUNT, COUNTA, COUNTBLANK, COUNTIF(S), DATE, DATEDIF, DAY, DAYS, DAYS360, EDATE, EOMONTH, EXACT, EXP, FIND, FLOOR,
|
|
115
115
|
HLOOKUP, IF, IFERROR, IFNA, IFS, INDEX, INT, IS* (BLANK, ERR, ERROR, LOGICAL, NA, NUMBER, TEXT), LEFT, LEN, LN, LOG,
|
|
116
|
-
LOG10, LOWER, MATCH, MAX, MAXIFS, MEDIAN, MID, MIN, MINIFS, MOD, MONTH, MROUND, NA, NOT, OR, PI, POWER, PRODUCT,
|
|
116
|
+
LOG10, LOOKUP, LOWER, MATCH, MAX, MAXIFS, MEDIAN, MID, MIN, MINIFS, MOD, MODE, MONTH, MROUND, NA, NOT, OR, PI, POWER, PRODUCT,
|
|
117
117
|
PROPER, REPLACE, REPT, RIGHT, ROUND(UP/DOWN), SEARCH, SIGN, SQRT, SUBSTITUTE, SUBTOTAL, SUM, SUMIF(S), SUMPRODUCT,
|
|
118
|
-
SWITCH, TEXT, TEXTJOIN, TRIM, TRUE/FALSE, TRUNC, UPPER, VALUE, VLOOKUP, WEEKDAY, XLOOKUP, XOR, YEAR.
|
|
118
|
+
SWITCH, TEXT, TEXTJOIN, TRIM, TRUE/FALSE, TRUNC, UPPER, VALUE, VLOOKUP, WEEKDAY, XLOOKUP, XOR, YEAR, YEARFRAC.
|
|
119
119
|
|
|
120
120
|
**Speed** (Windows 11 laptop, measured): a 200,000-formula workbook (four formula columns over 50,000 rows, one of them
|
|
121
121
|
a running total) converts and verifies in 26 s, all PASS. 50,000 rows of VLOOKUP into a 1,000-row table: 4.6 s.
|
|
@@ -126,13 +126,23 @@ COUNTIFS/SUMIFS with a criterion from the row run at the same speed.
|
|
|
126
126
|
cellproof prefers an honest UNRESOLVED or FAIL to a guess. Known limits, each reported with a reason:
|
|
127
127
|
|
|
128
128
|
- **File formats:** `.xlsx` and `.xlsm` only (no `.xls`, `.xlsb`, `.ods`). Macros (VBA) are ignored.
|
|
129
|
-
- **
|
|
130
|
-
|
|
131
|
-
|
|
132
|
-
|
|
129
|
+
- **Array constants** (`{0,50,100}`) work wherever a range is taken (`SUM`, `MATCH`, `LOOKUP`, `VLOOKUP`, `INDEX`...);
|
|
130
|
+
where one value is expected (`A1*{1,2}`) they need array evaluation and are UNRESOLVED.
|
|
131
|
+
- **Not supported:** array formulas and dynamic arrays, volatile functions (`NOW`, `TODAY`, `RAND`, `OFFSET`, and
|
|
132
|
+
`INDIRECT` of text built while the sheet runs; `INDIRECT("Sales")` is fine), functions outside the list above,
|
|
133
|
+
external workbooks, structured table references (`Table1[@Col]`), INDEX used as a reference, the 1904 date system in
|
|
134
|
+
TEXT date formats.
|
|
135
|
+
- **Ranges where one value is expected** (`=IF(E$3:E$41="F",B$3:B$41)`) take the formula's own row or column, as
|
|
136
|
+
Excel does (implicit intersection). A range that IF/IFERROR/CHOOSE hands to another function
|
|
137
|
+
(`MAX(IF(c,A1:A9,0))`, where Excel uses the whole range) is UNRESOLVED.
|
|
138
|
+
- **Approximate lookups on unsorted data** (LOOKUP, and MATCH/VLOOKUP/HLOOKUP with match type 1/-1): UNRESOLVED, because Excel's
|
|
133
139
|
answer then depends on its binary search.
|
|
134
|
-
- **Text to number:** only plain numbers convert (`"3"`, `" -1.5e3 "`). Excel
|
|
135
|
-
|
|
140
|
+
- **Text to number:** only plain numbers convert (`"3"`, `" -1.5e3 "`). Text Excel reads with the computer's language
|
|
141
|
+
settings (`"1,000"`, `"$5"`, `"50%"`, `"1/2/2020"`, `"2:00 PM"`, `"Jan 5"`) makes the cell UNRESOLVED, because the
|
|
142
|
+
file does not record those settings.
|
|
143
|
+
- **Who saved the file:** the report warns when it was last saved by another program (LibreOffice, a script): the
|
|
144
|
+
saved values then follow that program's rules. In a file Excel for Mac calculated, `CHAR` above 127 is UNRESOLVED
|
|
145
|
+
(the Mac character set differs).
|
|
136
146
|
- **Some function options:** SUBTOTAL 101-111 (hidden rows), DATEDIF units MD/YM/YD, WEEKDAY types 11-17, TEXT formats
|
|
137
147
|
outside the supported table, COUNT or MEDIAN of values typed into the formula, criteria taken from an empty cell.
|
|
138
148
|
- **Floating point:** a running total becomes a fast window function only when Excel's saved values prove the different
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# cellproof: progress
|
|
2
2
|
|
|
3
|
-
Last updated: 2026-10-
|
|
3
|
+
Last updated: 2026-10-20.
|
|
4
4
|
|
|
5
5
|
## State
|
|
6
6
|
|
|
@@ -16,11 +16,21 @@ Everything below is on `main`, built test-first, each piece reviewed by a fresh
|
|
|
16
16
|
| Slice 2 | a region with a cell between its columns splits instead of dissolving; running totals (`=C1+B2`, `=SUM($B$2:B2)`) become window functions |
|
|
17
17
|
| Lookups row-wise | VLOOKUP, HLOOKUP, MATCH, INDEX, XLOOKUP pasted down compile into the region's statement (one join each instead of one query per cell) |
|
|
18
18
|
| Tokenize once | a formula pasted down a column is tokenized once; the next rows' text is predicted and only re-tokenized when it differs |
|
|
19
|
+
| Implicit intersection | a one-column range where one value is expected takes the formula's row; `INDIRECT` of fixed text; MODE (found in a user's workbook) |
|
|
19
20
|
| Criteria row-wise | the *IF(S) family with a criterion from the row (`=COUNTIFS($E:$E,">"&A2,...)`) compiles into the region's statement too |
|
|
20
21
|
| Open source | MIT license, README with limits, SECURITY.md, CI on Windows/macOS/Linux x Python 3.10-3.13 |
|
|
21
22
|
|
|
22
|
-
Tests: about
|
|
23
|
-
purpose). Public on GitHub (one-commit history start). 0.1.0 was released as `unexcel`; 0.2.0
|
|
23
|
+
Tests: about 437 (dbt build included), plus 3 workbooks saved by real Excel that all PASS (4 cells UNRESOLVED on
|
|
24
|
+
purpose). Public on GitHub (one-commit history start). 0.1.0 was released as `unexcel`; 0.2.0 (2026-10-19) and 0.2.1 (2026-10-20) are on PyPI as `cellproof` (see CHANGELOG.md).
|
|
25
|
+
A user's practice workbook: 3 PASS / 47 UNRESOLVED -> 48 PASS / 0 FAIL / 2 UNRESOLVED.
|
|
26
|
+
|
|
27
|
+
## Public test workbooks (2026-10-19)
|
|
28
|
+
|
|
29
|
+
`python tests/corpus.py DIR` runs the 85 .xlsx/.xlsm test files of pycel, formulas and xlcalculator (Excel-saved,
|
|
30
|
+
except 8 LibreOffice ones): 6,874 PASS, 10 FAIL (all in LibreOffice files, flagged in their reports), 16,764
|
|
31
|
+
UNRESOLVED, no crashes (was 4,693 PASS before LOOKUP, YEARFRAC and array constants). The UNRESOLVED are mostly functions those files test on purpose: CONVERT, YEARFRAC,
|
|
32
|
+
DEC2HEX/BIN/OCT, array formulas and `{1,2}` constants, CODE/UNICODE, LOOKUP. For business workbooks the ones that
|
|
33
|
+
matter are array formulas, array constants, LOOKUP and YEARFRAC. Findings are tests in `tests/test_corpus_findings.py`.
|
|
24
34
|
|
|
25
35
|
## Measured speed (Windows 11 laptop)
|
|
26
36
|
|
|
@@ -70,6 +80,8 @@ What Excel taught us (each has a test in `tests/test_excel_findings.py`):
|
|
|
70
80
|
|
|
71
81
|
## Next
|
|
72
82
|
|
|
73
|
-
1.
|
|
74
|
-
|
|
75
|
-
|
|
83
|
+
1. Array formulas and array evaluation (`SUM(A1:A10*B1:B10)`, `MAX(IF(...))`, `A1*{1,2}`). LOOKUP, YEARFRAC and
|
|
84
|
+
array constants where a range is taken are done (unreleased).
|
|
85
|
+
2. Test on more real analyst workbooks (one user workbook found 4 gaps the 85 public files did not).
|
|
86
|
+
3. Read time (openpyxl loads the workbook twice) and the per-cell dependency loop.
|
|
87
|
+
4. Plan 3: AI retry loop for failed cells (it sees formula structure only, never values).
|
|
@@ -76,12 +76,12 @@ reader -> regions -> dependency graph -> compiler -> verifier -> dbt emitter ->
|
|
|
76
76
|
code.
|
|
77
77
|
6. **Emit** the dbt project and the report.
|
|
78
78
|
|
|
79
|
-
|
|
79
|
+
96 Excel functions are supported: ABS, AND, AVERAGE, AVERAGEIF(S), CEILING, CHAR, CHOOSE, CLEAN, CONCAT(ENATE),
|
|
80
80
|
COUNT, COUNTA, COUNTBLANK, COUNTIF(S), DATE, DATEDIF, DAY, DAYS, DAYS360, EDATE, EOMONTH, EXACT, EXP, FIND, FLOOR,
|
|
81
81
|
HLOOKUP, IF, IFERROR, IFNA, IFS, INDEX, INT, IS* (BLANK, ERR, ERROR, LOGICAL, NA, NUMBER, TEXT), LEFT, LEN, LN, LOG,
|
|
82
|
-
LOG10, LOWER, MATCH, MAX, MAXIFS, MEDIAN, MID, MIN, MINIFS, MOD, MONTH, MROUND, NA, NOT, OR, PI, POWER, PRODUCT,
|
|
82
|
+
LOG10, LOOKUP, LOWER, MATCH, MAX, MAXIFS, MEDIAN, MID, MIN, MINIFS, MOD, MODE, MONTH, MROUND, NA, NOT, OR, PI, POWER, PRODUCT,
|
|
83
83
|
PROPER, REPLACE, REPT, RIGHT, ROUND(UP/DOWN), SEARCH, SIGN, SQRT, SUBSTITUTE, SUBTOTAL, SUM, SUMIF(S), SUMPRODUCT,
|
|
84
|
-
SWITCH, TEXT, TEXTJOIN, TRIM, TRUE/FALSE, TRUNC, UPPER, VALUE, VLOOKUP, WEEKDAY, XLOOKUP, XOR, YEAR.
|
|
84
|
+
SWITCH, TEXT, TEXTJOIN, TRIM, TRUE/FALSE, TRUNC, UPPER, VALUE, VLOOKUP, WEEKDAY, XLOOKUP, XOR, YEAR, YEARFRAC.
|
|
85
85
|
|
|
86
86
|
**Speed** (Windows 11 laptop, measured): a 200,000-formula workbook (four formula columns over 50,000 rows, one of them
|
|
87
87
|
a running total) converts and verifies in 26 s, all PASS. 50,000 rows of VLOOKUP into a 1,000-row table: 4.6 s.
|
|
@@ -92,13 +92,23 @@ COUNTIFS/SUMIFS with a criterion from the row run at the same speed.
|
|
|
92
92
|
cellproof prefers an honest UNRESOLVED or FAIL to a guess. Known limits, each reported with a reason:
|
|
93
93
|
|
|
94
94
|
- **File formats:** `.xlsx` and `.xlsm` only (no `.xls`, `.xlsb`, `.ods`). Macros (VBA) are ignored.
|
|
95
|
-
- **
|
|
96
|
-
|
|
97
|
-
|
|
98
|
-
|
|
95
|
+
- **Array constants** (`{0,50,100}`) work wherever a range is taken (`SUM`, `MATCH`, `LOOKUP`, `VLOOKUP`, `INDEX`...);
|
|
96
|
+
where one value is expected (`A1*{1,2}`) they need array evaluation and are UNRESOLVED.
|
|
97
|
+
- **Not supported:** array formulas and dynamic arrays, volatile functions (`NOW`, `TODAY`, `RAND`, `OFFSET`, and
|
|
98
|
+
`INDIRECT` of text built while the sheet runs; `INDIRECT("Sales")` is fine), functions outside the list above,
|
|
99
|
+
external workbooks, structured table references (`Table1[@Col]`), INDEX used as a reference, the 1904 date system in
|
|
100
|
+
TEXT date formats.
|
|
101
|
+
- **Ranges where one value is expected** (`=IF(E$3:E$41="F",B$3:B$41)`) take the formula's own row or column, as
|
|
102
|
+
Excel does (implicit intersection). A range that IF/IFERROR/CHOOSE hands to another function
|
|
103
|
+
(`MAX(IF(c,A1:A9,0))`, where Excel uses the whole range) is UNRESOLVED.
|
|
104
|
+
- **Approximate lookups on unsorted data** (LOOKUP, and MATCH/VLOOKUP/HLOOKUP with match type 1/-1): UNRESOLVED, because Excel's
|
|
99
105
|
answer then depends on its binary search.
|
|
100
|
-
- **Text to number:** only plain numbers convert (`"3"`, `" -1.5e3 "`). Excel
|
|
101
|
-
|
|
106
|
+
- **Text to number:** only plain numbers convert (`"3"`, `" -1.5e3 "`). Text Excel reads with the computer's language
|
|
107
|
+
settings (`"1,000"`, `"$5"`, `"50%"`, `"1/2/2020"`, `"2:00 PM"`, `"Jan 5"`) makes the cell UNRESOLVED, because the
|
|
108
|
+
file does not record those settings.
|
|
109
|
+
- **Who saved the file:** the report warns when it was last saved by another program (LibreOffice, a script): the
|
|
110
|
+
saved values then follow that program's rules. In a file Excel for Mac calculated, `CHAR` above 127 is UNRESOLVED
|
|
111
|
+
(the Mac character set differs).
|
|
102
112
|
- **Some function options:** SUBTOTAL 101-111 (hidden rows), DATEDIF units MD/YM/YD, WEEKDAY types 11-17, TEXT formats
|
|
103
113
|
outside the supported table, COUNT or MEDIAN of values typed into the formula, criteria taken from an empty cell.
|
|
104
114
|
- **Floating point:** a running total becomes a fast window function only when Excel's saved values prove the different
|
|
@@ -0,0 +1,266 @@
|
|
|
1
|
+
# cellproof: the whole journey, from idea to today
|
|
2
|
+
|
|
3
|
+
This file tells the story of the project from the first design (2026-10-11) to now (2026-10-20): what was built, in
|
|
4
|
+
what order, why, what broke, and what was learned. The technical details of each step live in `DESIGN.md`,
|
|
5
|
+
`docs/superpowers/specs/`, `docs/superpowers/plans/`, `PROGRESS.md` and `CHANGELOG.md`; this is the readable summary.
|
|
6
|
+
|
|
7
|
+
---
|
|
8
|
+
|
|
9
|
+
## 1. The idea (2026-10-11)
|
|
10
|
+
|
|
11
|
+
**Problem.** A lot of business logic lives in Excel workbooks that one person understands. Moving that logic into a
|
|
12
|
+
database is slow and risky, because nobody can prove that the new numbers equal the old ones. Existing tools copy the
|
|
13
|
+
*data* out of Excel; none translated the *formulas*.
|
|
14
|
+
|
|
15
|
+
**Idea.** A command-line tool that:
|
|
16
|
+
|
|
17
|
+
1. reads every formula of an `.xlsx`,
|
|
18
|
+
2. translates each one into SQL,
|
|
19
|
+
3. runs that SQL on DuckDB (a fast local database),
|
|
20
|
+
4. compares every result with the value Excel itself saved in the file.
|
|
21
|
+
|
|
22
|
+
Excel's saved values are a free answer key. Every formula cell gets one verdict:
|
|
23
|
+
|
|
24
|
+
| Verdict | Meaning |
|
|
25
|
+
|---|---|
|
|
26
|
+
| PASS | the SQL gives the value Excel saved |
|
|
27
|
+
| FAIL | the SQL gives a different value; the report shows both |
|
|
28
|
+
| UNRESOLVED | the tool could not translate it exactly, and says why. It never guesses. |
|
|
29
|
+
|
|
30
|
+
The output is a **dbt project** (the standard way data teams ship SQL), so the result is something a team can use.
|
|
31
|
+
|
|
32
|
+
**Rules decided on day one** (they still hold):
|
|
33
|
+
|
|
34
|
+
- Never guess a value. Anything not translated exactly is UNRESOLVED with a reason a user understands.
|
|
35
|
+
- A false PASS is the worst possible bug. When unsure, give an honest FAIL or UNRESOLVED.
|
|
36
|
+
- Real Excel is the ground truth, not our idea of what Excel does.
|
|
37
|
+
- Cell values never leave the machine. No network calls with workbook data.
|
|
38
|
+
- Only two runtime dependencies: `openpyxl` (reads the file) and `duckdb` (runs the SQL).
|
|
39
|
+
- Tests first, and measure before claiming speed.
|
|
40
|
+
|
|
41
|
+
The working name was **unexcel**, already known to clash with an R package.
|
|
42
|
+
|
|
43
|
+
---
|
|
44
|
+
|
|
45
|
+
## 2. Timeline at a glance
|
|
46
|
+
|
|
47
|
+
| Date | Step | Result |
|
|
48
|
+
|---|---|---|
|
|
49
|
+
| 2026-10-11 | Design + Plan 1: the core | reader, parser, references, dependency order, SQL compiler, verifier, dbt output, report, CLI |
|
|
50
|
+
| 2026-10-11 | Plan 2a: scale and Excel errors | batches (no crash at ~1k formulas), Excel error codes, safety limits |
|
|
51
|
+
| 2026-10-12 | Plan 2b: function library | 93 Excel functions, run-time types, Excel's conversions |
|
|
52
|
+
| 2026-10-14 | First check against real Excel | 3 workbooks saved by real Excel: every cell PASS (4 UNRESOLVED on purpose) |
|
|
53
|
+
| 2026-10-14 | Plan 2c: speed in cell mode | faster dependency order, range index, smarter batches |
|
|
54
|
+
| 2026-10-16 | Plan 2d: table mode | a column of one pasted-down formula becomes one readable SQL statement |
|
|
55
|
+
| 2026-10-18 | Slice 2 | regions split instead of dissolving; running totals as window functions |
|
|
56
|
+
| 2026-10-18 | Open-source release 0.1.0 | public GitHub repo (fresh history), MIT license, CI, PyPI as `unexcel` |
|
|
57
|
+
| 2026-10-19 | Lookups and criteria row-wise | VLOOKUP/INDEX/MATCH/XLOOKUP and COUNTIFS/SUMIFS in table mode |
|
|
58
|
+
| 2026-10-19 | Faster compile | a pasted-down formula is tokenized once |
|
|
59
|
+
| 2026-10-19 | Renamed to **cellproof**, 0.2.0 | new name, new PyPI package, repo renamed |
|
|
60
|
+
| 2026-10-19 | 85 public test workbooks | FAIL 229 -> 10 (all 10 in LibreOffice-saved files), 0 crashes |
|
|
61
|
+
| 2026-10-19 | First real user workbook | 3 PASS / 47 UNRESOLVED -> 48 PASS / 0 FAIL / 2 UNRESOLVED |
|
|
62
|
+
|
|
63
|
+
---
|
|
64
|
+
|
|
65
|
+
## 3. Step by step
|
|
66
|
+
|
|
67
|
+
### 3.1 Plan 1: the core (2026-10-11)
|
|
68
|
+
|
|
69
|
+
Built test-first, one module at a time:
|
|
70
|
+
|
|
71
|
+
- **Reader**: loads the workbook twice with openpyxl, once for formulas and once for the values Excel saved.
|
|
72
|
+
- **Parser**: formula text to a tree, with Excel's operator precedence (`-2^2` is 4 in Excel).
|
|
73
|
+
- **References**: `A1`, `Sheet!B2:C9`, whole columns, defined names.
|
|
74
|
+
- **Dependency order**: every cell after the cells it reads; circular references detected.
|
|
75
|
+
- **Compiler (cell mode)**: each formula becomes a SQL expression.
|
|
76
|
+
- **Verifier**: runs everything on DuckDB and compares with Excel's saved values.
|
|
77
|
+
- **Emitter**: writes the dbt project; `--dbt` also runs `dbt build`.
|
|
78
|
+
- **Report**: `REPORT.md` and `report.json`.
|
|
79
|
+
- **CLI**: `unexcel convert workbook.xlsx`.
|
|
80
|
+
|
|
81
|
+
The first end-to-end test was a small commission workbook.
|
|
82
|
+
|
|
83
|
+
### 3.2 Plan 2a: scale and Excel errors (2026-10-11)
|
|
84
|
+
|
|
85
|
+
- The first version crashed around 1,000 formulas (one giant SQL statement). Fix: run formulas in **batches** over one
|
|
86
|
+
shared cells table.
|
|
87
|
+
- Excel **error codes** (`#DIV/0!`, `#N/A`, `#VALUE!`...) now flow through operators and functions, and must match
|
|
88
|
+
exactly.
|
|
89
|
+
- Safety: a deeply nested formula or one whose SQL would grow exponentially becomes UNRESOLVED instead of crashing or
|
|
90
|
+
eating memory; a styled cell on the last row no longer costs seconds or gigabytes.
|
|
91
|
+
- Optional fallback answer key (the `formulas` library) for files with no saved values, always labelled as such.
|
|
92
|
+
- A 200,000-formula benchmark was added: 390 s at that point.
|
|
93
|
+
|
|
94
|
+
### 3.3 Plan 2b: the function library (2026-10-12)
|
|
95
|
+
|
|
96
|
+
- **93 Excel functions** (SUM, IF, VLOOKUP, XLOOKUP, INDEX/MATCH, SUMIFS, TEXT, dates, ...).
|
|
97
|
+
- Run-time types: a cell's type can be known only when the SQL runs ("any").
|
|
98
|
+
- One criteria engine for the whole COUNTIF/SUMIF/AVERAGEIF(S) family (`">"&B1`, wildcards, `"<>"`).
|
|
99
|
+
|
|
100
|
+
### 3.4 First check against real Excel (2026-10-14)
|
|
101
|
+
|
|
102
|
+
Three workbooks were saved by real Microsoft Excel and checked cell by cell. Excel taught us things no document says,
|
|
103
|
+
each now a test (`tests/test_excel_findings.py`):
|
|
104
|
+
|
|
105
|
+
- COUNTIF `"<>5"` skips only the number 5 (the text "5" is counted).
|
|
106
|
+
- DAYS360, `TEXT(-0.001,"0.00")`, `MOD(1,0.1)`, dates inside 1900, `(-8)^(1/3)`, error order in `SUM(H3,1/0)`.
|
|
107
|
+
- Approximate MATCH/VLOOKUP on unsorted data depends on Excel's binary search, so it is UNRESOLVED.
|
|
108
|
+
|
|
109
|
+
Result: every cell of the three workbooks PASS, 4 UNRESOLVED on purpose.
|
|
110
|
+
|
|
111
|
+
### 3.5 Plan 2c: speed in cell mode (2026-10-14)
|
|
112
|
+
|
|
113
|
+
Measured first, then four changes: a Fenwick tree for dependency order without expanding ranges, a range index,
|
|
114
|
+
batches closed by SQL size instead of cell count, smaller dbt models.
|
|
115
|
+
|
|
116
|
+
### 3.6 Plan 2d: table mode (2026-10-16)
|
|
117
|
+
|
|
118
|
+
Most real sheets repeat one formula down a column (`=B2*C2`, `=B3*C3`, ...). Such a column is now a **region**:
|
|
119
|
+
compiled once into one readable SQL statement (a "wide model" with real column names), instead of one expression per
|
|
120
|
+
cell. Hard parts solved along the way: safe SQL names from headers, regions that would create false cycles, proving
|
|
121
|
+
that table mode gives exactly the same answers as cell mode (a test rule since then).
|
|
122
|
+
|
|
123
|
+
### 3.7 Slice 2 (2026-10-18)
|
|
124
|
+
|
|
125
|
+
- A region with another cell between its columns is split instead of thrown away.
|
|
126
|
+
- Running totals (`=C1+B2`, `=SUM($B$2:B2)`) become SQL window functions, but only when Excel's saved values prove
|
|
127
|
+
the different addition order cannot change the answer. A ledger whose balance cancels back to zero stays in
|
|
128
|
+
Excel's exact order.
|
|
129
|
+
- 200k-formula benchmark: **390 s -> 31 s**, all PASS.
|
|
130
|
+
|
|
131
|
+
### 3.8 Open-source release 0.1.0 (2026-10-18)
|
|
132
|
+
|
|
133
|
+
- MIT license, README with an honest limits section, SECURITY.md, CI on Windows/macOS/Linux x Python 3.10-3.13.
|
|
134
|
+
- Security review: zip bombs refused, crafted numbers beyond Excel's range UNRESOLVED, workbook text only reaches SQL
|
|
135
|
+
as escaped literals.
|
|
136
|
+
- **Privacy**: the public repository was created with a fresh one-commit history and a no-reply author address.
|
|
137
|
+
- Published to PyPI as `unexcel` 0.1.0 with trusted publishing (no token stored anywhere).
|
|
138
|
+
|
|
139
|
+
### 3.9 Lookups and criteria in table mode (2026-10-19)
|
|
140
|
+
|
|
141
|
+
- **Problem**: a VLOOKUP pasted down a column was still one query per cell. DuckDB could not plan the SQL form the
|
|
142
|
+
lookups used inside a region.
|
|
143
|
+
- **Fix**: every lookup became an aggregate (`arg_min` by position) over a table that does not depend on the row,
|
|
144
|
+
which DuckDB turns into one join for the whole column.
|
|
145
|
+
- 1,000 VLOOKUPs: verify **5.7 s -> 0.1 s**. 50,000 VLOOKUPs: all PASS.
|
|
146
|
+
- Same idea for COUNTIFS/SUMIFS with a criterion from the row:
|
|
147
|
+
- 1,000 COUNTIFS: verify **29 s -> 0.1 s**.
|
|
148
|
+
|
|
149
|
+
### 3.10 Faster compile (2026-10-19)
|
|
150
|
+
|
|
151
|
+
Profiling showed the formula tokenizer ran twice per cell and was most of the compile time. Now the first cell of a
|
|
152
|
+
pasted-down column is tokenized and the next rows are predicted by shifting row numbers; a cell is tokenized again only
|
|
153
|
+
when the prediction does not match its real text.
|
|
154
|
+
|
|
155
|
+
- 50k VLOOKUPs: compile **6.1 s -> 1.8 s**.
|
|
156
|
+
- 200k benchmark: **31 s -> 26 s**.
|
|
157
|
+
|
|
158
|
+
### 3.11 Renamed to cellproof, 0.2.0 (2026-10-19)
|
|
159
|
+
|
|
160
|
+
- `unexcel` clashed with an R package. Candidates were checked on PyPI, CRAN, npm and GitHub.
|
|
161
|
+
- **cellproof** was free everywhere and says what is unique: every cell is proven against Excel.
|
|
162
|
+
- Package, command, import, dbt project and GitHub repository were renamed (old links redirect).
|
|
163
|
+
- 0.2.0 was published to PyPI as `cellproof`. The old `unexcel` 0.1.0 stays on PyPI under the old name.
|
|
164
|
+
|
|
165
|
+
### 3.12 Testing on 85 public workbooks (2026-10-19)
|
|
166
|
+
|
|
167
|
+
The tool was run on every test workbook of three other Excel formula engines (pycel, formulas, xlcalculator), most
|
|
168
|
+
of them saved by real Excel. First run: 1 crash, 229 FAIL. Each FAIL was traced to its cause:
|
|
169
|
+
|
|
170
|
+
- **Bugs in cellproof, fixed**:
|
|
171
|
+
- a name defined as a formula crashed the reader;
|
|
172
|
+
- Excel's 29 February 1900 and date-formatted negative numbers were read wrong;
|
|
173
|
+
- escaped control characters in text were not decoded;
|
|
174
|
+
- AND/OR must skip text and empty cells;
|
|
175
|
+
- an empty result of IF/IFERROR is 0, not "";
|
|
176
|
+
- EDATE/EOMONTH/MROUND/VALUE refuse TRUE/FALSE;
|
|
177
|
+
- error order in SUMPRODUCT;
|
|
178
|
+
- PROPER of another cell built SQL DuckDB rejected.
|
|
179
|
+
- **Not bugs, now handled honestly**:
|
|
180
|
+
- 8 files were saved by LibreOffice, not Excel, so their saved values follow LibreOffice; the report now warns.
|
|
181
|
+
- One file was calculated by Excel for Mac, where CHAR above 127 gives other characters; now UNRESOLVED.
|
|
182
|
+
- Text like `"1/2/2020"` or `"1,000"` is read by Excel using the computer's language settings, which the file does
|
|
183
|
+
not record; now UNRESOLVED instead of a wrong answer.
|
|
184
|
+
- One of my own earlier assumptions (`AND(TRUE,"0")` is `#VALUE!`) was wrong: real Excel says TRUE. The test was
|
|
185
|
+
corrected to match Excel, never the other way round.
|
|
186
|
+
|
|
187
|
+
Result: **FAIL 229 -> 10** (all 10 in the LibreOffice files), **0 crashes**. `python tests/corpus.py DIR` repeats it.
|
|
188
|
+
|
|
189
|
+
### 3.13 The first real user workbook (2026-10-19)
|
|
190
|
+
|
|
191
|
+
A practice workbook full of array-style formulas: **3 PASS, 0 FAIL, 47 UNRESOLVED**. Added:
|
|
192
|
+
|
|
193
|
+
- **Implicit intersection**: `=IF(E$3:E$41="F",B$3:B$41)` uses the formula's own row, as Excel does (40 cells).
|
|
194
|
+
- **`INDIRECT("English")`** of fixed text is just the named range.
|
|
195
|
+
- **MODE**.
|
|
196
|
+
|
|
197
|
+
The first version of implicit intersection gave 89 for `=MAX(IF(E3:E41="F",math,0))` where Excel saved 99: the
|
|
198
|
+
verifier caught it as a FAIL. Investigation showed Excel passes the whole `math` range through IF into MAX, so that
|
|
199
|
+
case is now honestly UNRESOLVED. The same investigation found an older bug: a bare word like `=i` with no matching
|
|
200
|
+
name was read as the whole column I.
|
|
201
|
+
|
|
202
|
+
Result: **48 PASS, 0 FAIL, 2 UNRESOLVED** (one real array formula, one range passed through IF).
|
|
203
|
+
|
|
204
|
+
---
|
|
205
|
+
|
|
206
|
+
## 4. How it works now (short version)
|
|
207
|
+
|
|
208
|
+
```
|
|
209
|
+
workbook.xlsx
|
|
210
|
+
| reader: formulas + Excel's saved values (and which program saved the file)
|
|
211
|
+
v
|
|
212
|
+
parse every formula -> find columns of one pasted-down formula (regions)
|
|
213
|
+
|
|
|
214
|
+
v
|
|
215
|
+
dependency order (regions as one block) -> SQL: one statement per region, small batches for the rest
|
|
216
|
+
|
|
|
217
|
+
v
|
|
218
|
+
run on DuckDB -> compare every cell with Excel's saved value -> PASS / FAIL / UNRESOLVED
|
|
219
|
+
|
|
|
220
|
+
v
|
|
221
|
+
REPORT.md + report.json + a dbt project (wide readable models per region)
|
|
222
|
+
```
|
|
223
|
+
|
|
224
|
+
---
|
|
225
|
+
|
|
226
|
+
## 5. Where things stand (2026-10-20)
|
|
227
|
+
|
|
228
|
+
| | |
|
|
229
|
+
|---|---|
|
|
230
|
+
| Public repository | github.com/Nikhil414/cellproof (MIT, CI green) |
|
|
231
|
+
| PyPI | `pip install cellproof` (0.2.1) |
|
|
232
|
+
| Tests | 425 passing (dbt build included) |
|
|
233
|
+
| Functions | 94 Excel functions |
|
|
234
|
+
| Real Excel checks | 3 own workbooks all PASS; 85 public workbooks: only FAILs in LibreOffice files; 1 user workbook 48/0/2 |
|
|
235
|
+
|
|
236
|
+
**Speed** (Windows 11 laptop):
|
|
237
|
+
|
|
238
|
+
| Workbook | Time |
|
|
239
|
+
|---|---|
|
|
240
|
+
| 200k formulas (4 columns x 50k rows, a running total) | 26 s, all PASS (was 390 s) |
|
|
241
|
+
| 50k VLOOKUPs into a 1,000-row table | 4.6 s, all PASS |
|
|
242
|
+
| 1,000 COUNTIFS with a criterion from the row | verify 0.1 s (was 29 s) |
|
|
243
|
+
|
|
244
|
+
---
|
|
245
|
+
|
|
246
|
+
## 6. What we learned
|
|
247
|
+
|
|
248
|
+
1. **Real Excel is the only reliable reference.** Several "obvious" Excel behaviours turned out wrong once checked
|
|
249
|
+
against files Excel saved.
|
|
250
|
+
2. **The answer key catches our own mistakes.** Every wrong translation so far showed up as a FAIL, including two of
|
|
251
|
+
my own wrong fixes; none slipped through as a false PASS.
|
|
252
|
+
3. **Real files find what tests do not.** One user workbook needed three new features; 85 public workbooks found
|
|
253
|
+
eight bugs.
|
|
254
|
+
4. **Not every FAIL is our bug.** A file can be saved by LibreOffice, calculated on a Mac, or depend on language
|
|
255
|
+
settings. The tool now says so instead of blaming itself or guessing.
|
|
256
|
+
5. **Measure, then optimize.** Every speed-up came from a profile or a benchmark, not from a hunch.
|
|
257
|
+
|
|
258
|
+
---
|
|
259
|
+
|
|
260
|
+
## 7. What is next
|
|
261
|
+
|
|
262
|
+
1. Release 0.2.1 with the fixes since 0.2.0.
|
|
263
|
+
2. Launch prep: a short demo and a sample report on the README front page; then posts on LinkedIn and Twitter/X.
|
|
264
|
+
3. Array formulas and `{1,2}` array constants (the biggest remaining gap for real workbooks), LOOKUP, YEARFRAC.
|
|
265
|
+
4. More real workbooks from users.
|
|
266
|
+
5. Plan 3: an AI retry for cells that fail, which sees only the formula structure, never the values.
|