sqlmath 2026.8.31__cp315-cp315-win_amd64.whl
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
|
@@ -0,0 +1,696 @@
|
|
|
1
|
+
Metadata-Version: 2.1
|
|
2
|
+
Name: sqlmath
|
|
3
|
+
Version: 2026.8.31
|
|
4
|
+
Summary: sqlite for data-science
|
|
5
|
+
Author: Kai Zhu
|
|
6
|
+
Requires-Python: >=3.10
|
|
7
|
+
License: MIT
|
|
8
|
+
Classifier: Development Status :: 4 - Beta
|
|
9
|
+
Classifier: Intended Audience :: Developers
|
|
10
|
+
Classifier: Intended Audience :: Science/Research
|
|
11
|
+
Classifier: License :: OSI Approved :: MIT License
|
|
12
|
+
Classifier: Operating System :: MacOS
|
|
13
|
+
Classifier: Operating System :: Microsoft :: Windows
|
|
14
|
+
Classifier: Operating System :: POSIX :: Linux
|
|
15
|
+
Classifier: Programming Language :: C
|
|
16
|
+
Classifier: Programming Language :: JavaScript
|
|
17
|
+
Classifier: Programming Language :: Python
|
|
18
|
+
Classifier: Topic :: Database
|
|
19
|
+
Classifier: Topic :: Scientific/Engineering
|
|
20
|
+
Classifier: Topic :: Software Development
|
|
21
|
+
Project-URL: Homepage, https://github.com/sqlmath/sqlmath
|
|
22
|
+
Project-URL: Changelog, https://github.com/sqlmath/sqlmath/blob/master/CHANGELOG.md
|
|
23
|
+
License-File: LICENSE
|
|
24
|
+
Description-Content-Type: text/markdown
|
|
25
|
+
|
|
26
|
+
# sqlmath
|
|
27
|
+
|
|
28
|
+
**SQLite with data-science superpowers.** Run statistics, machine learning, and analytics directly in SQL queries — from Python, JavaScript, or the browser.
|
|
29
|
+
|
|
30
|
+
[](https://pypi.org/project/sqlmath/)
|
|
31
|
+
[](https://www.npmjs.com/package/sqlmath)<!--no-validate-->
|
|
32
|
+
[](https://sqlmath.github.io/sqlmath/index.html)
|
|
33
|
+
[](https://sqlmath.github.io/sqlmath/apidoc.html)
|
|
34
|
+
|
|
35
|
+
# Status
|
|
36
|
+
|
|
37
|
+
| Branch | [master<br>(v2026.8.31)](https://github.com/sqlmath/sqlmath/tree/master) | [beta<br>(Web Demo)](https://github.com/sqlmath/sqlmath/tree/beta) | [alpha<br>(Development)](https://github.com/sqlmath/sqlmath/tree/alpha) |
|
|
38
|
+
|--:|:--:|:--:|:--:|
|
|
39
|
+
| CI | [](https://github.com/sqlmath/sqlmath/actions?query=branch%3Amaster) | [](https://github.com/sqlmath/sqlmath/actions?query=branch%3Abeta) | [](https://github.com/sqlmath/sqlmath/actions?query=branch%3Aalpha) |
|
|
40
|
+
| Coverage | [](https://sqlmath.github.io/sqlmath/branch-master/.artifact/coverage/index.html) | [](https://sqlmath.github.io/sqlmath/branch-beta/.artifact/coverage/index.html) | [](https://sqlmath.github.io/sqlmath/branch-alpha/.artifact/coverage/index.html) |
|
|
41
|
+
| Demo | [<img src="https://sqlmath.github.io/sqlmath/asset_image_github_brands.svg" height="32">](https://sqlmath.github.io/sqlmath/branch-master/index.html) | [<img src="https://sqlmath.github.io/sqlmath/asset_image_github_brands.svg" height="32">](https://sqlmath.github.io/sqlmath/branch-beta/index.html) | [<img src="https://sqlmath.github.io/sqlmath/asset_image_github_brands.svg" height="32">](https://sqlmath.github.io/sqlmath/branch-alpha/index.html) |
|
|
42
|
+
| Artifacts | [<img src="https://sqlmath.github.io/sqlmath/asset_image_folder_open_solid.svg" height="30">](https://github.com/sqlmath/sqlmath/tree/gh-pages/branch-master/.artifact) | [<img src="https://sqlmath.github.io/sqlmath/asset_image_folder_open_solid.svg" height="30">](https://github.com/sqlmath/sqlmath/tree/gh-pages/branch-beta/.artifact) | [<img src="https://sqlmath.github.io/sqlmath/asset_image_folder_open_solid.svg" height="30">](https://github.com/sqlmath/sqlmath/tree/gh-pages/branch-alpha/.artifact) |
|
|
43
|
+
|
|
44
|
+
|
|
45
|
+
<br><br>
|
|
46
|
+
# Table of Contents
|
|
47
|
+
|
|
48
|
+
1. [Web Demo](#web-demo)
|
|
49
|
+
|
|
50
|
+
2. [Why sqlmath?](#why-sqlmath)
|
|
51
|
+
|
|
52
|
+
3. [Quickstart](#quickstart)
|
|
53
|
+
- [Python](#python)
|
|
54
|
+
- [JavaScript (Node.js)](#javascript-nodejs)
|
|
55
|
+
- [Browser (WebAssembly)](#browser-webassembly)
|
|
56
|
+
|
|
57
|
+
4. [Built-in SQL Functions](#built-in-sql-functions)
|
|
58
|
+
- [Math](#math)
|
|
59
|
+
- [Statistics (Aggregate)](#statistics-aggregate)
|
|
60
|
+
- [Statistics (Scalar)](#statistics-scalar)
|
|
61
|
+
- [Arrays](#arrays)
|
|
62
|
+
- [Date/Time](#datetime)
|
|
63
|
+
- [Time Series / Signal Processing](#time-series--signal-processing)
|
|
64
|
+
- [Compression](#compression)
|
|
65
|
+
- [Type Casting](#type-casting)
|
|
66
|
+
- [Cryptography](#cryptography)
|
|
67
|
+
|
|
68
|
+
5. [Machine Learning with LightGBM](#machine-learning-with-lightgbm)
|
|
69
|
+
- [Training from a Table](#training-from-a-table)
|
|
70
|
+
- [Prediction](#prediction)
|
|
71
|
+
- [Real-World Example: Credit Card Fraud Detection](#real-world-example-credit-card-fraud-detection)
|
|
72
|
+
- [LightGBM Functions](#lightgbm-functions)
|
|
73
|
+
|
|
74
|
+
6. [Use Cases](#use-cases)
|
|
75
|
+
- [Financial Data Analysis](#financial-data-analysis)
|
|
76
|
+
- [Embedded ML Pipelines](#embedded-ml-pipelines)
|
|
77
|
+
- [Data Compression & Hashing](#data-compression--hashing)
|
|
78
|
+
|
|
79
|
+
7. [Why sqlmath vs Alternatives?](#why-sqlmath-vs-alternatives)
|
|
80
|
+
|
|
81
|
+
8. [Python API Reference](#python-api-reference)
|
|
82
|
+
- [db_exec()](#db_exec)
|
|
83
|
+
|
|
84
|
+
9. [Node.js API Reference](#nodejs-api-reference)
|
|
85
|
+
|
|
86
|
+
10. [Platform Notes](#platform-notes)
|
|
87
|
+
|
|
88
|
+
11. [Building from Source](#building-from-source)
|
|
89
|
+
- [Prerequisites](#prerequisites)
|
|
90
|
+
- [Build](#build)
|
|
91
|
+
- [Run Tests](#run-tests)
|
|
92
|
+
- [Serve Demo Locally](#serve-demo-locally)
|
|
93
|
+
|
|
94
|
+
12. [Package Listing](#package-listing)
|
|
95
|
+
|
|
96
|
+
13. [Changelog](#changelog)
|
|
97
|
+
|
|
98
|
+
14. [License](#license)
|
|
99
|
+
|
|
100
|
+
15. [Devops Instruction](#devops-instruction)
|
|
101
|
+
- [python pypi publish](#python-pypi-publish)
|
|
102
|
+
- [sqlite upgrade](#sqlite-upgrade)
|
|
103
|
+
|
|
104
|
+
|
|
105
|
+
<br><br>
|
|
106
|
+
# Web Demo
|
|
107
|
+
- https://sqlmath.github.io/sqlmath/index.html
|
|
108
|
+
|
|
109
|
+
[](https://sqlmath.github.io/sqlmath/index.html)
|
|
110
|
+
|
|
111
|
+
|
|
112
|
+
<br><br>
|
|
113
|
+
# Why sqlmath?
|
|
114
|
+
|
|
115
|
+
SQLite is everywhere — it's the most deployed database in the world. But it lacks the statistical and ML functions needed for data-science. sqlmath fixes that.
|
|
116
|
+
|
|
117
|
+
**What you get:**
|
|
118
|
+
- **20+ built-in functions** for math, statistics, arrays, dates, and compression
|
|
119
|
+
- **LightGBM integration** — train and predict ML models directly in SQL
|
|
120
|
+
- **Zero dependencies** — single binary, no external libraries needed
|
|
121
|
+
- **Multi-platform** — Python, Node.js, browser (WebAssembly)
|
|
122
|
+
|
|
123
|
+
|
|
124
|
+
<br><br>
|
|
125
|
+
# Quickstart
|
|
126
|
+
|
|
127
|
+
|
|
128
|
+
<br><br>
|
|
129
|
+
### Python
|
|
130
|
+
|
|
131
|
+
```shell
|
|
132
|
+
pip install sqlmath
|
|
133
|
+
```
|
|
134
|
+
|
|
135
|
+
```python
|
|
136
|
+
from sqlmath import db_open, db_exec, db_close
|
|
137
|
+
|
|
138
|
+
# Open an in-memory database
|
|
139
|
+
db = db_open(":memory:")
|
|
140
|
+
|
|
141
|
+
# Create sample data
|
|
142
|
+
db_exec(db=db, sql="""
|
|
143
|
+
CREATE TABLE prices (symbol TEXT, price REAL);
|
|
144
|
+
INSERT INTO prices VALUES ('AAPL', 150.0), ('GOOGL', 140.0), ('MSFT', 380.0);
|
|
145
|
+
""")
|
|
146
|
+
|
|
147
|
+
# Use built-in statistical functions
|
|
148
|
+
result = db_exec(db=db, sql="""
|
|
149
|
+
SELECT
|
|
150
|
+
AVG(price) AS mean_price,
|
|
151
|
+
MEDIAN2(price) AS median_price,
|
|
152
|
+
STDEV(price) AS stdev_price
|
|
153
|
+
FROM prices
|
|
154
|
+
""")
|
|
155
|
+
print(result)
|
|
156
|
+
# [[{'mean_price': 223.33, 'median_price': 150.0, 'stdev_price': 135.77}]]
|
|
157
|
+
|
|
158
|
+
db_close(db)
|
|
159
|
+
```
|
|
160
|
+
|
|
161
|
+
|
|
162
|
+
<br><br>
|
|
163
|
+
### JavaScript (Node.js)
|
|
164
|
+
|
|
165
|
+
```shell
|
|
166
|
+
npm install sqlmath
|
|
167
|
+
```
|
|
168
|
+
|
|
169
|
+
```javascript
|
|
170
|
+
import { dbOpenAsync, dbExecAsync, dbCloseAsync } from "sqlmath";
|
|
171
|
+
|
|
172
|
+
const db = await dbOpenAsync({ filename: ":memory:" });
|
|
173
|
+
|
|
174
|
+
await dbExecAsync({
|
|
175
|
+
db,
|
|
176
|
+
sql: `
|
|
177
|
+
CREATE TABLE prices (symbol TEXT, price REAL);
|
|
178
|
+
INSERT INTO prices VALUES ('AAPL', 150.0), ('GOOGL', 140.0), ('MSFT', 380.0);
|
|
179
|
+
`
|
|
180
|
+
});
|
|
181
|
+
|
|
182
|
+
const result = await dbExecAsync({
|
|
183
|
+
db,
|
|
184
|
+
sql: "SELECT AVG(price) AS mean_price FROM prices"
|
|
185
|
+
});
|
|
186
|
+
console.log(result);
|
|
187
|
+
|
|
188
|
+
await dbCloseAsync(db);
|
|
189
|
+
```
|
|
190
|
+
|
|
191
|
+
|
|
192
|
+
<br><br>
|
|
193
|
+
### Browser (WebAssembly)
|
|
194
|
+
|
|
195
|
+
Try it live: **[sqlmath.github.io/sqlmath](https://sqlmath.github.io/sqlmath/index.html)**
|
|
196
|
+
|
|
197
|
+
|
|
198
|
+
<br><br>
|
|
199
|
+
# Built-in SQL Functions
|
|
200
|
+
|
|
201
|
+
|
|
202
|
+
<br><br>
|
|
203
|
+
### Math
|
|
204
|
+
|
|
205
|
+
| Function | Description |
|
|
206
|
+
|----------|-------------|
|
|
207
|
+
| `cot(x)` | Cotangent |
|
|
208
|
+
| `coth(x)` | Hyperbolic cotangent |
|
|
209
|
+
| `fmod(x, y)` | Floating-point modulo |
|
|
210
|
+
| `squared(x)` | Square of x |
|
|
211
|
+
| `squaredwithsign(x)` | Square preserving sign (negative input → negative output) |
|
|
212
|
+
| `sqrtwithsign(x)` | Square root preserving sign |
|
|
213
|
+
| `normalizewithsqrt(x)` | Normalize using square root |
|
|
214
|
+
| `normalizewithsquared(x)` | Normalize using square |
|
|
215
|
+
|
|
216
|
+
|
|
217
|
+
<br><br>
|
|
218
|
+
### Statistics (Aggregate)
|
|
219
|
+
|
|
220
|
+
| Function | Description |
|
|
221
|
+
|----------|-------------|
|
|
222
|
+
| `MEDIAN2(x)` | Median value (aggregate) |
|
|
223
|
+
| `PERCENTILE(x, p)` | Percentile at p (aggregate) |
|
|
224
|
+
| `QUANTILE(x, q)` | Quantile at q (aggregate) |
|
|
225
|
+
| `STDEV(x)` | Standard deviation (sample, window-capable) |
|
|
226
|
+
|
|
227
|
+
|
|
228
|
+
<br><br>
|
|
229
|
+
### Statistics (Scalar)
|
|
230
|
+
|
|
231
|
+
| Function | Description |
|
|
232
|
+
|----------|-------------|
|
|
233
|
+
| `marginoferror95(p, n)` | 95% margin of error: sqrt(p*(1-p)/n) |
|
|
234
|
+
| `random1()` | Random float in [0, 1) |
|
|
235
|
+
|
|
236
|
+
|
|
237
|
+
<br><br>
|
|
238
|
+
### Arrays
|
|
239
|
+
|
|
240
|
+
| Function | Description |
|
|
241
|
+
|----------|-------------|
|
|
242
|
+
| `doublearray_array(...)` | Create double array from values |
|
|
243
|
+
| `doublearray_extract(arr, i)` | Extract element at index |
|
|
244
|
+
| `doublearray_jsonfrom(json)` | Create array from JSON |
|
|
245
|
+
| `doublearray_jsonto(arr)` | Convert array to JSON |
|
|
246
|
+
|
|
247
|
+
|
|
248
|
+
<br><br>
|
|
249
|
+
### Date/Time
|
|
250
|
+
|
|
251
|
+
sqlmath extends SQLite's date functions with integer-format conversions (YYYYMMDDHHMMSS).
|
|
252
|
+
|
|
253
|
+
|
|
254
|
+
<br><br>
|
|
255
|
+
### Time Series / Signal Processing
|
|
256
|
+
|
|
257
|
+
| Function | Description |
|
|
258
|
+
|----------|-------------|
|
|
259
|
+
| `WIN_SINEFIT2(...)` | Fit sine wave to data (aggregate window function) |
|
|
260
|
+
| `SINEFIT_REFITLAST(...)` | Refit sine wave with new data point |
|
|
261
|
+
|
|
262
|
+
|
|
263
|
+
<br><br>
|
|
264
|
+
### Compression
|
|
265
|
+
|
|
266
|
+
| Function | Description |
|
|
267
|
+
|----------|-------------|
|
|
268
|
+
| `GZIP_COMPRESS(blob)` | Compress BLOB with gzip |
|
|
269
|
+
| `GZIP_UNCOMPRESS(blob)` | Decompress gzip BLOB |
|
|
270
|
+
|
|
271
|
+
> **Note:** Input must be BLOB. Use `CAST(text AS BLOB)` for text data.
|
|
272
|
+
|
|
273
|
+
|
|
274
|
+
<br><br>
|
|
275
|
+
### Type Casting
|
|
276
|
+
|
|
277
|
+
| Function | Description |
|
|
278
|
+
|----------|-------------|
|
|
279
|
+
| `castrealornull(x)` | Cast to REAL or NULL |
|
|
280
|
+
| `castrealorzero(x)` | Cast to REAL or 0.0 |
|
|
281
|
+
| `casttextorempty(x)` | Cast to TEXT or '' |
|
|
282
|
+
| `roundorzero(x, n)` | Round or return 0 |
|
|
283
|
+
|
|
284
|
+
|
|
285
|
+
<br><br>
|
|
286
|
+
### Cryptography
|
|
287
|
+
|
|
288
|
+
| Function | Description |
|
|
289
|
+
|----------|-------------|
|
|
290
|
+
| `sha256(data)` | SHA-256 hash (returns BLOB) |
|
|
291
|
+
|
|
292
|
+
```sql
|
|
293
|
+
SELECT HEX(sha256('hello')) AS hash;
|
|
294
|
+
-- 2CF24DBA5FB0A30E26E83B2AC5B9E29E1B161E5C1FA7425E73043362938B9824
|
|
295
|
+
```
|
|
296
|
+
|
|
297
|
+
|
|
298
|
+
<br><br>
|
|
299
|
+
# Machine Learning with LightGBM
|
|
300
|
+
|
|
301
|
+
sqlmath embeds [LightGBM](https://lightgbm.readthedocs.io/) for gradient boosting directly in SQL queries. Train models on your data without leaving SQL — no data shuffle to Python needed.
|
|
302
|
+
|
|
303
|
+
> **Python users:** You must call `lgbm_dlopen()` before using any `lgbm_*` functions. See [Platform Notes](#platform-notes) for details.
|
|
304
|
+
|
|
305
|
+
|
|
306
|
+
<br><br>
|
|
307
|
+
### Training from a Table
|
|
308
|
+
|
|
309
|
+
`LGBM_TRAINFROMTABLE` is an **aggregate function** — it consumes rows like `SUM()` or `AVG()`, but outputs a trained model BLOB.
|
|
310
|
+
|
|
311
|
+
```sql
|
|
312
|
+
-- Create a table to store the model
|
|
313
|
+
CREATE TABLE model_store (model BLOB);
|
|
314
|
+
|
|
315
|
+
-- Train directly from your data table
|
|
316
|
+
INSERT INTO model_store(model)
|
|
317
|
+
SELECT
|
|
318
|
+
LGBM_TRAINFROMTABLE(
|
|
319
|
+
-- Training parameters
|
|
320
|
+
(
|
|
321
|
+
'objective=binary'
|
|
322
|
+
|| ' learning_rate=0.1'
|
|
323
|
+
|| ' metric=auc'
|
|
324
|
+
|| ' num_leaves=31'
|
|
325
|
+
|| ' verbosity=0'
|
|
326
|
+
),
|
|
327
|
+
50, -- num_iterations
|
|
328
|
+
10, -- eval_step (early stopping check interval)
|
|
329
|
+
'max_bin=15', -- data parameters
|
|
330
|
+
NULL, -- reference dataset (NULL for training set)
|
|
331
|
+
-- Columns: first column MUST be the label
|
|
332
|
+
label, feature1, feature2, feature3, feature4
|
|
333
|
+
)
|
|
334
|
+
FROM training_data;
|
|
335
|
+
```
|
|
336
|
+
|
|
337
|
+
|
|
338
|
+
<br><br>
|
|
339
|
+
### Prediction
|
|
340
|
+
|
|
341
|
+
```sql
|
|
342
|
+
-- Predict on new data using the stored model
|
|
343
|
+
SELECT
|
|
344
|
+
id,
|
|
345
|
+
LGBM_PREDICTFORTABLE(
|
|
346
|
+
(SELECT model FROM model_store),
|
|
347
|
+
0, -- predict_type (0 = normal probability)
|
|
348
|
+
0, -- start_iteration
|
|
349
|
+
50, -- num_iterations
|
|
350
|
+
'', -- prediction parameters
|
|
351
|
+
feature1, feature2, feature3, feature4
|
|
352
|
+
) AS prediction
|
|
353
|
+
FROM test_data;
|
|
354
|
+
```
|
|
355
|
+
|
|
356
|
+
|
|
357
|
+
<br><br>
|
|
358
|
+
### Real-World Example: Credit Card Fraud Detection
|
|
359
|
+
|
|
360
|
+
From the [Kaggle notebook](https://www.kaggle.com/code/kaizhu256/sql-is-all-you-need) — training on 284,807 transactions with 0.17% fraud rate:
|
|
361
|
+
|
|
362
|
+
```python
|
|
363
|
+
from sqlmath import db_open, db_exec, db_table_import
|
|
364
|
+
|
|
365
|
+
db = db_open("fraud.sqlite")
|
|
366
|
+
|
|
367
|
+
# Import Kaggle credit card dataset
|
|
368
|
+
db_table_import(db=db, filename="creditcard.csv", table_name="transactions", mode="csv")
|
|
369
|
+
|
|
370
|
+
# Split 80/20 train/test
|
|
371
|
+
db_exec(db=db, sql="""
|
|
372
|
+
CREATE TABLE train AS SELECT * FROM transactions WHERE RANDOM() % 5 != 0;
|
|
373
|
+
CREATE TABLE test AS SELECT * FROM transactions WHERE RANDOM() % 5 = 0;
|
|
374
|
+
""")
|
|
375
|
+
|
|
376
|
+
# Train with automatic class imbalance handling
|
|
377
|
+
db_exec(db=db, sql="""
|
|
378
|
+
CREATE TABLE model_store (model BLOB);
|
|
379
|
+
|
|
380
|
+
INSERT INTO model_store(model)
|
|
381
|
+
SELECT
|
|
382
|
+
LGBM_TRAINFROMTABLE(
|
|
383
|
+
(
|
|
384
|
+
'objective=binary'
|
|
385
|
+
|| ' learning_rate=0.05'
|
|
386
|
+
|| ' metric=auc'
|
|
387
|
+
|| ' is_unbalance=true' -- Auto-weight minority class
|
|
388
|
+
|| ' num_leaves=31'
|
|
389
|
+
|| ' verbosity=0'
|
|
390
|
+
),
|
|
391
|
+
100, -- num_iterations
|
|
392
|
+
10, -- eval_step
|
|
393
|
+
'max_bin=255', -- data_params
|
|
394
|
+
NULL, -- reference
|
|
395
|
+
-- First column = label, rest = features (V1-V28 + Amount)
|
|
396
|
+
Class, V1, V2, V3, V4, V5, V6, V7, V8, V9, V10,
|
|
397
|
+
V11, V12, V13, V14, V15, V16, V17, V18, V19, V20,
|
|
398
|
+
V21, V22, V23, V24, V25, V26, V27, V28, Amount
|
|
399
|
+
)
|
|
400
|
+
FROM train;
|
|
401
|
+
""")
|
|
402
|
+
```
|
|
403
|
+
|
|
404
|
+
|
|
405
|
+
<br><br>
|
|
406
|
+
### LightGBM Functions
|
|
407
|
+
|
|
408
|
+
| Function | Type | Description |
|
|
409
|
+
|----------|------|-------------|
|
|
410
|
+
| `lgbm_dlopen(path)` | scalar | Load LightGBM library (required in Python) |
|
|
411
|
+
| `LGBM_TRAINFROMTABLE(params, n_iter, eval, data_params, ref, label, ...)` | aggregate | Train model from table columns |
|
|
412
|
+
| `LGBM_TRAINFROMFILE(params, n_iter, eval, train_file, data_params, test_file)` | scalar | Train from LibSVM file |
|
|
413
|
+
| `LGBM_PREDICTFORTABLE(model, type, start, n, params, ...)` | window | Predict per row |
|
|
414
|
+
| `LGBM_PREDICTFORFILE(model, type, start, n, params, file, header, out)` | scalar | Predict and save to file |
|
|
415
|
+
| `lgbm_extract(result, key)` | scalar | Extract value from result |
|
|
416
|
+
|
|
417
|
+
**Complete Example:**
|
|
418
|
+
|
|
419
|
+
[](https://www.kaggle.com/code/kaizhu256/sql-is-all-you-need)
|
|
420
|
+
|
|
421
|
+
Full notebook with fraud detection, intraday trading signals, and model persistence.
|
|
422
|
+
|
|
423
|
+
**Kaggle Environment:**
|
|
424
|
+
- Linux x64, Python 3.12, Node.js 24
|
|
425
|
+
- Datasets: [Credit Card Fraud](https://www.kaggle.com/datasets/mlg-ulb/creditcardfraud) (284K transactions), SPY intraday OHLCV
|
|
426
|
+
- sqlmath installs via `pip install sqlmath==2026.8.31`
|
|
427
|
+
|
|
428
|
+
|
|
429
|
+
<br><br>
|
|
430
|
+
# Use Cases
|
|
431
|
+
|
|
432
|
+
|
|
433
|
+
<br><br>
|
|
434
|
+
### Financial Data Analysis
|
|
435
|
+
|
|
436
|
+
```python
|
|
437
|
+
from sqlmath import db_open, db_exec, db_close, db_table_import
|
|
438
|
+
|
|
439
|
+
db = db_open(":memory:")
|
|
440
|
+
|
|
441
|
+
# Import OHLCV data
|
|
442
|
+
db_table_import(db=db, table_name="prices", filename="prices.csv", mode="csv")
|
|
443
|
+
|
|
444
|
+
# Calculate rolling statistics in SQL
|
|
445
|
+
result = db_exec(db=db, sql="""
|
|
446
|
+
SELECT
|
|
447
|
+
date,
|
|
448
|
+
close,
|
|
449
|
+
AVG(close) OVER (ORDER BY date ROWS 20 PRECEDING) AS sma_20,
|
|
450
|
+
STDEV(close) OVER (ORDER BY date ROWS 20 PRECEDING) AS volatility,
|
|
451
|
+
PERCENTILE(volume, 50) OVER (ORDER BY date ROWS 5 PRECEDING) AS median_volume
|
|
452
|
+
FROM prices
|
|
453
|
+
ORDER BY date DESC
|
|
454
|
+
LIMIT 10
|
|
455
|
+
""")
|
|
456
|
+
|
|
457
|
+
db_close(db)
|
|
458
|
+
```
|
|
459
|
+
|
|
460
|
+
|
|
461
|
+
<br><br>
|
|
462
|
+
### Embedded ML Pipelines
|
|
463
|
+
|
|
464
|
+
Train and deploy models without data movement:
|
|
465
|
+
|
|
466
|
+
```python
|
|
467
|
+
# 1. Load data into SQLite
|
|
468
|
+
db_table_import(db=db, table_name="features", filename="training_data.csv", mode="csv")
|
|
469
|
+
|
|
470
|
+
# 2. Train model in-database
|
|
471
|
+
db_exec(db=db, sql="""
|
|
472
|
+
CREATE TABLE models AS
|
|
473
|
+
SELECT LGBM_TRAINFROMTABLE(...) AS model FROM features
|
|
474
|
+
""")
|
|
475
|
+
|
|
476
|
+
# 3. Score new data in-database
|
|
477
|
+
db_exec(db=db, sql="""
|
|
478
|
+
SELECT id, LGBM_PREDICTFORTABLE(model, ...) AS score
|
|
479
|
+
FROM new_data, models
|
|
480
|
+
""")
|
|
481
|
+
|
|
482
|
+
# 4. Export results
|
|
483
|
+
db_file_save(db=db, filename="scored_data.sqlite")
|
|
484
|
+
```
|
|
485
|
+
|
|
486
|
+
|
|
487
|
+
<br><br>
|
|
488
|
+
### Data Compression & Hashing
|
|
489
|
+
|
|
490
|
+
```sql
|
|
491
|
+
-- Compress large text fields
|
|
492
|
+
UPDATE documents SET
|
|
493
|
+
content_compressed = GZIP_COMPRESS(CAST(content AS BLOB));
|
|
494
|
+
|
|
495
|
+
-- Generate content hashes for deduplication
|
|
496
|
+
SELECT HEX(sha256(content)) AS content_hash, COUNT(*)
|
|
497
|
+
FROM documents
|
|
498
|
+
GROUP BY content_hash
|
|
499
|
+
HAVING COUNT(*) > 1;
|
|
500
|
+
```
|
|
501
|
+
|
|
502
|
+
|
|
503
|
+
<br><br>
|
|
504
|
+
# Why sqlmath vs Alternatives?
|
|
505
|
+
|
|
506
|
+
| Feature | sqlmath | pandas | DuckDB | sqlite3 |
|
|
507
|
+
|---------|---------|--------|--------|---------|
|
|
508
|
+
| In-database ML | ✅ LightGBM | ❌ | ❌ | ❌ |
|
|
509
|
+
| Zero data shuffle | ✅ | ❌ | ✅ | ✅ |
|
|
510
|
+
| Statistical functions | ✅ 20+ | ✅ | ✅ | ❌ |
|
|
511
|
+
| Browser support | ✅ WASM | ❌ | ❌ | ❌ |
|
|
512
|
+
| Single file deployment | ✅ | ❌ | ✅ | ✅ |
|
|
513
|
+
| Memory efficiency | ✅ Streaming | ❌ In-memory | ✅ | ✅ |
|
|
514
|
+
|
|
515
|
+
**Choose sqlmath when:**
|
|
516
|
+
- You need ML training/inference inside the database
|
|
517
|
+
- Your data is already in SQLite
|
|
518
|
+
- You want browser-based analytics (WebAssembly)
|
|
519
|
+
- You need statistical functions beyond basic SQL
|
|
520
|
+
|
|
521
|
+
**Choose pandas when:**
|
|
522
|
+
- You need complex data transformations
|
|
523
|
+
- Interactive exploration with rich visualization
|
|
524
|
+
- Your workflow is Python-centric
|
|
525
|
+
|
|
526
|
+
**Choose DuckDB when:**
|
|
527
|
+
- You need fast analytical queries on large datasets
|
|
528
|
+
- You want pandas DataFrame interop
|
|
529
|
+
- OLAP workloads are primary
|
|
530
|
+
|
|
531
|
+
|
|
532
|
+
<br><br>
|
|
533
|
+
# Python API Reference
|
|
534
|
+
|
|
535
|
+
```python
|
|
536
|
+
from sqlmath import (
|
|
537
|
+
db_open, # Open database
|
|
538
|
+
db_close, # Close database
|
|
539
|
+
db_exec, # Execute SQL, return results
|
|
540
|
+
db_exec_and_return_lastblob, # Execute SQL, return last blob
|
|
541
|
+
db_file_load, # Load database from file
|
|
542
|
+
db_file_save, # Save database to file
|
|
543
|
+
db_table_import, # Import data into table
|
|
544
|
+
db_noop, # No-op for testing
|
|
545
|
+
)
|
|
546
|
+
```
|
|
547
|
+
|
|
548
|
+
|
|
549
|
+
<br><br>
|
|
550
|
+
### db_exec()
|
|
551
|
+
|
|
552
|
+
```python
|
|
553
|
+
result = db_exec(
|
|
554
|
+
db=db, # Database connection
|
|
555
|
+
sql="SELECT ...", # SQL statement(s)
|
|
556
|
+
bind_list=[...], # Bind parameters (list or dict)
|
|
557
|
+
response_type=None, # None (default) returns list-of-dicts
|
|
558
|
+
# "list" | "lastblob" | "arraybuffer"
|
|
559
|
+
)
|
|
560
|
+
# Default: [[{'col1': val1, 'col2': val2}, ...]]
|
|
561
|
+
```
|
|
562
|
+
|
|
563
|
+
|
|
564
|
+
<br><br>
|
|
565
|
+
# Node.js API Reference
|
|
566
|
+
- https://sqlmath.github.io/sqlmath/apidoc.html
|
|
567
|
+
|
|
568
|
+
[](https://sqlmath.github.io/sqlmath/apidoc.html)
|
|
569
|
+
|
|
570
|
+
|
|
571
|
+
<br><br>
|
|
572
|
+
# Platform Notes
|
|
573
|
+
|
|
574
|
+
The JavaScript (Node.js) binding is more mature than Python. Key differences:
|
|
575
|
+
|
|
576
|
+
| Feature | Node.js | Python |
|
|
577
|
+
|---------|---------|--------|
|
|
578
|
+
| Async vs sync api | **async only:** `result = await dbExecAsync({...})` | **sync only:** `result = db_exec(...)` |
|
|
579
|
+
| Connection pooling | ✅ `db = await dbOpenAsync({threadCount: 4, ...})` | ❌ `db = db_open(...)` |
|
|
580
|
+
| Type hints | N/A | ❌ Not yet |
|
|
581
|
+
| Context manager | N/A | ❌ `with db_open()` not supported |
|
|
582
|
+
|
|
583
|
+
> **Note:** The Python wrapper is a work in progress. Contributions welcome!
|
|
584
|
+
|
|
585
|
+
|
|
586
|
+
<br><br>
|
|
587
|
+
# Building from Source
|
|
588
|
+
|
|
589
|
+
|
|
590
|
+
<br><br>
|
|
591
|
+
### Prerequisites
|
|
592
|
+
|
|
593
|
+
- Node.js 24+
|
|
594
|
+
- Python 3.12+
|
|
595
|
+
- C compiler (gcc, clang, or MSVC)
|
|
596
|
+
|
|
597
|
+
|
|
598
|
+
<br><br>
|
|
599
|
+
### Build
|
|
600
|
+
|
|
601
|
+
```shell
|
|
602
|
+
#!/bin/sh
|
|
603
|
+
|
|
604
|
+
# git clone sqlmath repo
|
|
605
|
+
git clone https://github.com/sqlmath/sqlmath --branch=beta --single-branch
|
|
606
|
+
cd sqlmath
|
|
607
|
+
|
|
608
|
+
# Build native binary
|
|
609
|
+
npm run test2
|
|
610
|
+
|
|
611
|
+
# Build WebAssembly (optional)
|
|
612
|
+
sh jslint_ci.sh shCiBuildWasm
|
|
613
|
+
```
|
|
614
|
+
|
|
615
|
+
|
|
616
|
+
<br><br>
|
|
617
|
+
### Run Tests
|
|
618
|
+
|
|
619
|
+
```shell
|
|
620
|
+
# Full test suite (includes full clean-build)
|
|
621
|
+
npm run test2
|
|
622
|
+
|
|
623
|
+
# Full test suite (includes partial re-build of modified files)
|
|
624
|
+
npm run test
|
|
625
|
+
|
|
626
|
+
# Quick test (skip build)
|
|
627
|
+
npm run test --fast
|
|
628
|
+
```
|
|
629
|
+
|
|
630
|
+
|
|
631
|
+
<br><br>
|
|
632
|
+
### Serve Demo Locally
|
|
633
|
+
|
|
634
|
+
```shell
|
|
635
|
+
PORT=8080 sh jslint_ci.sh shHttpFileServer
|
|
636
|
+
# Open http://localhost:8080/index.html
|
|
637
|
+
```
|
|
638
|
+
|
|
639
|
+
|
|
640
|
+
<br><br>
|
|
641
|
+
# Package Listing
|
|
642
|
+

|
|
643
|
+
|
|
644
|
+
|
|
645
|
+
<br><br>
|
|
646
|
+
# Changelog
|
|
647
|
+
- [Full CHANGELOG.md](CHANGELOG.md)
|
|
648
|
+
|
|
649
|
+

|
|
650
|
+
|
|
651
|
+
|
|
652
|
+
<br><br>
|
|
653
|
+
# License
|
|
654
|
+
- [sqlite](https://github.com/sqlite/sqlite) is under [public domain](https://www.sqlite.org/copyright.html).
|
|
655
|
+
- [jslint](https://github.com/jslint-org/jslint) is under [Unlicense License](https://github.com/jslint-org/jslint/blob/master/LICENSE).
|
|
656
|
+
- [zlib](https://github.com/madler/zlib) is under [zlib License](https://github.com/madler/zlib/blob/v1.3.1/LICENSE).
|
|
657
|
+
- [cpplint.py](cpplint.py) is under [3-Clause BSD License](https://github.com/cpplint/cpplint/blob/2.0.0/LICENSE).
|
|
658
|
+
- [indent.exe](indent.exe) is under [GPLv3 License](https://www.gnu.org/licenses/gpl-3.0.txt)<!--no-validate-->.
|
|
659
|
+
- Everything else is under MIT License.
|
|
660
|
+
|
|
661
|
+
|
|
662
|
+
<br><br>
|
|
663
|
+
# Devops Instruction
|
|
664
|
+
|
|
665
|
+
|
|
666
|
+
<br><br>
|
|
667
|
+
### python pypi publish
|
|
668
|
+
```shell
|
|
669
|
+
python -m build
|
|
670
|
+
#
|
|
671
|
+
twine upload --repository testpypi dist/sqlmath-2026.8.31*
|
|
672
|
+
py -m pip install --index-url https://test.pypi.org/simple/ sqlmath==2026.8.31
|
|
673
|
+
#
|
|
674
|
+
twine upload dist/sqlmath-2026.8.31*
|
|
675
|
+
pip install sqlmath==2026.8.31
|
|
676
|
+
```
|
|
677
|
+
|
|
678
|
+
|
|
679
|
+
<br><br>
|
|
680
|
+
### sqlite upgrade
|
|
681
|
+
- goto https://www.sqlite.org/changes.html
|
|
682
|
+
```shell
|
|
683
|
+
(set -e
|
|
684
|
+
#
|
|
685
|
+
# lgbm
|
|
686
|
+
sh jslint_ci.sh shRollupUpgrade "v4.6.0" "v4.7.0" ".ci.sh sqlmath_base.h"
|
|
687
|
+
#
|
|
688
|
+
# sqlite
|
|
689
|
+
sh jslint_ci.sh shRollupUpgrade "3.50.4" "3.51.3" ".ci.sh sqlmath_external_sqlite.c"
|
|
690
|
+
sh jslint_ci.sh shRollupUpgrade "3500400" "3510300" ".ci.sh sqlmath_external_sqlite.c"
|
|
691
|
+
#
|
|
692
|
+
# shSqlmathUpdate
|
|
693
|
+
read -p "Press Enter to shSqlmathUpdate:"
|
|
694
|
+
sh jslint_ci.sh shSqlmathUpdate
|
|
695
|
+
)
|
|
696
|
+
```
|
|
@@ -0,0 +1,7 @@
|
|
|
1
|
+
sqlmath/__init__.py,sha256=xlTgS0WNDqgE0rVZyHvZ5sFZcXrc06F0eiSCM1-sU1w,26987
|
|
2
|
+
sqlmath/lib_lightgbm_win32_x64.dll,sha256=fjZtLknNBhqsOrIWdrL5mwx6dY3D6IjUsjgSrxt9MBw,3888128
|
|
3
|
+
sqlmath/_sqlmath.cp315-win_amd64.pyd,sha256=QVYcdt5bP3gYlslblEK8umARUf4Oe2ed0rm1hMpMllw,1978368
|
|
4
|
+
sqlmath-2026.8.31.dist-info/LICENSE,sha256=KhokKrD65FnQuqeH79rJDSG_VdYZxu6rNKSHSAxpbYI,1051
|
|
5
|
+
sqlmath-2026.8.31.dist-info/METADATA,sha256=Zf6fKe7bRV7FsC5-wqj28xirmkOoRJZOwMR2lVfSJ3Y,21663
|
|
6
|
+
sqlmath-2026.8.31.dist-info/WHEEL,sha256=xha6PB11F-ubdozD3QuRMmyYRnpRdp1k1zsxobjUUSk,96
|
|
7
|
+
sqlmath-2026.8.31.dist-info/RECORD,,
|