laketower 0.5.1__py3-none-any.whl → 0.6.5__py3-none-any.whl

{laketower-0.5.1.dist-info → laketower-0.6.5.dist-info}/METADATA

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: laketower
-Version: 0.5.1
+Version: 0.6.5
 Summary: Oversee your lakehouse
 Project-URL: Repository, https://github.com/datalpia/laketower
 Project-URL: Issues, https://github.com/datalpia/laketower/issues
@@ -22,21 +22,24 @@ Classifier: Topic :: Database
 Classifier: Topic :: Software Development
 Classifier: Topic :: Utilities
 Requires-Python: <3.14,>=3.10
+Requires-Dist: bleach
 Requires-Dist: deltalake<2,>=1
 Requires-Dist: duckdb
 Requires-Dist: fastapi
 Requires-Dist: jinja2!=3.1.5,>=3
-Requires-Dist: pandas
+Requires-Dist: markdown
 Requires-Dist: pyarrow!=19.0.0
 Requires-Dist: pydantic-settings>=2
 Requires-Dist: pydantic>=2
+Requires-Dist: python-multipart
 Requires-Dist: pyyaml
 Requires-Dist: rich
 Requires-Dist: sqlglot
+Requires-Dist: tzdata
 Requires-Dist: uvicorn
 Description-Content-Type: text/markdown
 
-# Laketower
+# 🗼 Laketower
 
 > Oversee your lakehouse
 
@@ -50,13 +53,16 @@ Utility application to explore and manage tables in your data lakehouse, especia
 ## Features
 
 - Delta Lake table format support
+- Remote tables support (S3, ADLS)
 - Inspect table metadata
 - Inspect table schema
 - Inspect table history
 - Get table statistics
+- Import data into a table from CSV files
 - View table content with a simple query builder
 - Query all registered tables with DuckDB SQL dialect
 - Execute saved queries
+- Export query results to CSV files
 - Static and versionable YAML configuration
 - Web application
 - CLI application
@@ -86,6 +92,11 @@ Laketower configuration is based on a static YAML configuration file allowing to
 Format:
 
 ```yaml
+settings:
+  max_query_rows: 1000
+  web:
+    hide_tables: false
+
 tables:
   - name: <table_name>
     uri: <local path to table>
@@ -94,12 +105,18 @@ tables:
 queries:
   - name: <query_name>
     title: <Query name>
+    description: <Query description>
+    parameters:
+      <param_name_1>:
+        default: <default_value>
     sql: <sql expression>
 ```
 
 Current limitations:
 
-- `tables.uri`: only local paths are allowed
+- `tables.uri`:
+    - Local paths are supported (`./path/to/table`, `/abs/path/to/table`, `file:///abs/path/to/table`)
+    - Remote paths to S3 (`s3://<bucket>/<path>`) and ADLS (`abfss://<container>/<path>`)
 - `tables.format`: only `delta` is allowed
 
 Example from the provided demo:
@@ -138,6 +155,103 @@ queries:
         day asc
 ```
 
+Support for environment variables substitution is also supported within the YAML
+configuration using a object containing a single key `env` with the name of the
+environment variable to be injected. The value of the variable can contain JSON
+and will be decoded in a best effort manner (default to string value). For instance:
+
+```yaml
+# export TABLE_URI=path/to/table
+
+tables:
+  - name: sample_table
+    uri:
+      env: TABLE_URI
+    format: delta
+```
+
+#### Remote S3 Tables
+
+Configuring S3 tables (AWS, MinIO, Cloudflare R2):
+
+```yaml
+tables:
+  - name: delta_table_s3
+    uri: s3://<bucket>/path/to/table
+    format: delta
+    connection:
+      s3:
+        s3_access_key_id: access-key-id
+        s3_secret_access_key: secret-access-key
+        s3_region: s3-region
+        s3_endpoint_url: http://s3.domain.com
+        s3_allow_http: false
+```
+
+Depending on your object storage location and configuration, one might have to
+set part or all the available `connection.s3` parameters. The only required ones
+are `s3_access_key_id` and `s3_secret_access_key`.
+
+Also as a security best practice, it is best not to write secrets directly in
+static configuration files, so one can use environment variables to all dynamic substitution,
+e.g.
+
+```yaml
+tables:
+  - name: delta_table_s3
+    uri: s3://<bucket>/path/to/table
+    format: delta
+    connection:
+      s3:
+        s3_access_key_id: access-key-id
+        s3_secret_access_key:
+          env: S3_SECRET_ACCESS_KEY
+        s3_region: s3-region
+        s3_endpoint_url: http://s3.domain.com
+        s3_allow_http: false
+```
+
+#### Remote ADLS Tables
+
+Configuring Azure ADLS tables:
+
+```yaml
+tables:
+  - name: delta_table_adls
+    uri: abfss://<container>/path/to/table
+    format: delta
+    connection:
+      adls:
+        adls_account_name: adls-account-name
+        adls_access_key: adls-access-key
+        adls_sas_key: adls-sas-key
+        adls_tenant_id: adls-tenant-id
+        adls_client_id: adls-client-id
+        adls_client_secret: adls-client-secret
+        azure_msi_endpoint: https://msi.azure.com
+        use_azure_cli: false
+```
+
+Depending on your object storage location and configuration, one might have to
+set part or all the available `connection.adls` parameters. The only required one
+is `adls_account_name`.
+
+Also as a security best practice, it is best not to write secrets directly in
+static configuration files, so one can use environment variables to all dynamic substitution,
+e.g.
+
+```yaml
+tables:
+  - name: delta_table_adls
+    uri: abfss://<container>/path/to/table
+    format: delta
+    connection:
+      adls:
+        adls_account_name: adls-account-name
+        adls_access_key:
+          env: ADLS_ACCESS_KEY
+```
+
 ### Web Application
 
 The easiest way to get started is to launch the Laketower web application:
@@ -152,6 +266,7 @@ $ laketower -c demo/laketower.yml web
 ![Laketower UI - Tables View](https://raw.githubusercontent.com/datalpia/laketower/refs/heads/main/docs/static/tables_view.png)
 ![Laketower UI - Tables Statistics](https://raw.githubusercontent.com/datalpia/laketower/refs/heads/main/docs/static/tables_statistics.png)
 ![Laketower UI - Tables History](https://raw.githubusercontent.com/datalpia/laketower/refs/heads/main/docs/static/tables_history.png)
+![Laketower UI - Tables Import](https://raw.githubusercontent.com/datalpia/laketower/refs/heads/main/docs/static/tables_import.png)
 ![Laketower UI - Tables Query](https://raw.githubusercontent.com/datalpia/laketower/refs/heads/main/docs/static/tables_query.png)
 ![Laketower UI - Queries View](https://raw.githubusercontent.com/datalpia/laketower/refs/heads/main/docs/static/queries_view.png)
 
@@ -321,6 +436,29 @@ $ laketower -c demo/laketower.yml tables statistics --version 0 weather
 └──────────────────────┴───────┴──────┴──────┴──────┴──────┘
 ```
 
+#### Import data into a given table
+
+Import a CSV dataset into a table in append mode:
+
+```bash
+$ laketower -c demo/laketower.yml tables import weather --file data.csv --mode append --format csv --delimiter ',' --encoding 'utf-8'
+```
+
+`--mode` argument can be one of:
+- `append`: append rows to the table (default)
+- `overwrite`: replace all rows with the ones from the input file
+
+`--format` argument can be one of:
+- `csv`: CSV file format (default)
+
+`--delimiter` argument can be:
+- Any single character (only valid for CSV file format)
+- Default is _comma_ (`','`)
+
+`--encoding` argument can be:
+- Any [standard Python encoding](https://docs.python.org/3/library/codecs.html#standard-encodings),
+- Default is `'utf-8'`
+
 #### View a given table
 
 Using a simple query builder, the content of a table can be displayed.
@@ -398,6 +536,32 @@ $ laketower -c demo/laketower.yml tables query "select date_trunc('day', time) a
 │ 2025-02-11 00:00:00+01:00 │ 4.833333373069763  │
 │ 2025-02-10 00:00:00+01:00 │ 2.1083333243926368 │
 └───────────────────────────┴────────────────────┘
+3 rows returned
+Execution time: 33.72ms
+```
+
+Use named parameters within a giving query (note: escape `$` prefixes properly!):
+
+```bash
+$ laketower -c demo/laketower.yml tables query "select date_trunc('day', time) as day, avg(temperature_2m) as mean_temperature from weather where day between \$start_date and \$end_date group by day order by day desc" -p start_date 2025-01-29 -p end_date 2025-01-31
+
+┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┓
+┃ day                       ┃ mean_temperature   ┃
+┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━┩
+│ 2025-01-31 00:00:00+01:00 │ 5.683333257834117  │
+│ 2025-01-30 00:00:00+01:00 │ 8.900000015894571  │
+│ 2025-01-29 00:00:00+01:00 │ 7.770833313465118  │
+└───────────────────────────┴────────────────────┘
+4 rows returned
+Execution time: 30.59ms
+```
+
+Export query results to CSV:
+
+```bash
+$ laketower -c demo/laketower.yml tables query --output results.csv "select date_trunc('day', time) as day, avg(temperature_2m) as mean_temperature from weather group by day order by day desc limit 3"
+
+Query results written to: results.csv
 ```
 
 #### List saved queries
@@ -437,10 +601,30 @@ $ laketower -c demo/laketower.yml queries view daily_avg_temperature
 │ 2025-02-11 00:00:00+01:00 │ 5.0             │
 │ 2025-02-12 00:00:00+01:00 │ 5.0             │
 └───────────────────────────┴─────────────────┘
+18 rows returned
+Execution time: 39.52ms
+```
+
+Executing a predefined query with parameters (here `start_date` and `end_date`):
+
+```bash
+$ laketower -c demo/laketower.yml queries view daily_avg_temperature_params -p start_date 2025-02-01 -p end_date 2025-02-05
+
+┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓
+┃ day                       ┃ avg_temperature ┃
+┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩
+│ 2025-02-01 00:00:00+01:00 │ 4.0             │
+│ 2025-02-02 00:00:00+01:00 │ 4.0             │
+│ 2025-02-03 00:00:00+01:00 │ 4.0             │
+│ 2025-02-04 00:00:00+01:00 │ 3.0             │
+│ 2025-02-05 00:00:00+01:00 │ 3.0             │
+└───────────────────────────┴─────────────────┘
+6 rows returned
+Execution time: 32.08ms
 ```
 
 ## License
 
 Licensed under [Apache License 2.0](LICENSE)
 
-Copyright (c) 2025 - present Romain Clement
+Copyright (c) 2025 - present Romain Clement / Datalpia

laketower-0.6.5.dist-info/RECORD

@@ -0,0 +1,35 @@
+laketower/__about__.py,sha256=KDgkBrBsBSUzbLgrOZ89YsNN06fU4j5bmcuEwo6q5pg,22
+laketower/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+laketower/__main__.py,sha256=czKxJKG8OfncnxWmpaOWx7b1JBwFnZNQi7wKSTncB4M,108
+laketower/cli.py,sha256=6jwvG2ZjNQNOJJS6fWzOcr8cp2u0VMgMuBd9vIIaS8o,18071
+laketower/config.py,sha256=9zY69hIDSYcNqZtWGu-Jv23BFmLtHh2p03DZq9-1s_c,3703
+laketower/tables.py,sha256=o5OxrSTamUFZC8j7qBnJWEAZ_9tG1sPzif2XFAJjIM0,11713
+laketower/web.py,sha256=mm-zNsi8mc_5ig8TWXt4p8k2GghFvVIIjpAguAGT6Nw,14696
+laketower/static/.gitkeep,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+laketower/static/datatables.bundle.js,sha256=k9QXhh0gBLtw5gJRJToDrgdNVPBWAKRdNfBmYvziuiA,814319
+laketower/static/datatables.js,sha256=lsBkx3Kt9GwGDZ97JEgDkwvfBlzK3THlN1K43iOrqvM,1857
+laketower/static/editor.bundle.js,sha256=Wa1bS0xWwKDjHKPTdXd-PX41JAGkCjd0VLBCF-SyXWk,1159343
+laketower/static/editor.js,sha256=C8saQJH68X7CtdRyBmvrQxsDJ013YWoMfWF-6hzf_5s,15870
+laketower/static/vendor/bootstrap/bootstrap.bundle.min.js,sha256=5P1JGBOIxI7FBAvT_mb1fCnI5n_NhQKzNUuW7Hq0fMc,80496
+laketower/static/vendor/bootstrap-icons/bootstrap-icons.min.css,sha256=pdY4ejLKO67E0CM2tbPtq1DJ3VGDVVdqAR6j3ZwdiE4,87008
+laketower/static/vendor/bootstrap-icons/fonts/bootstrap-icons.woff,sha256=9VUTt7WRy4SjuH_w406iTUgx1v7cIuVLkRymS1tUShU,180288
+laketower/static/vendor/bootstrap-icons/fonts/bootstrap-icons.woff2,sha256=bHVxA2ShylYEJncW9tKJl7JjGf2weM8R4LQqtm_y6mE,134044
+laketower/static/vendor/datatables.net-bs5/dataTables.bootstrap5.css,sha256=OPNPBvNyQahoUZ5gvqCCz9nPPIoUug2nn2_fpg9D8H0,21837
+laketower/static/vendor/datatables.net-columncontrol-bs5/columnControl.bootstrap5.min.css,sha256=JwBYwZbQtbi-PwceKQ4KeqQp6E0QnGzeUSbaX-26UZc,13574
+laketower/static/vendor/halfmoon/halfmoon.min.css,sha256=RjeFzczeuZHCyS-Gvz-kleETzBF_o84ZRHukze_yv6o,369168
+laketower/static/vendor/halfmoon/halfmoon.modern.css,sha256=DD6elX-jPmbFYPsGvzodUv2-9FHkxHlVtQi0_RJVULs,10576
+laketower/templates/_base.html,sha256=bRSdILczy9V05fMNu-iz30dLB6e14XFUhurBu2AlGK4,7065
+laketower/templates/index.html,sha256=dLF2Og0qgzBkvGyVRidRNzTv0u4o97ifOx1jVeig8Kg,59
+laketower/templates/queries/view.html,sha256=2vEqpOsgx6lPCYEcxW5VM60oERwayjrDlSaCznJ6MkY,3426
+laketower/templates/tables/_macros.html,sha256=sCI1TOFW0QA74oSXW87H6dNTudOs7n-FretnTPFcRh4,1174
+laketower/templates/tables/history.html,sha256=a5GBLXCiLlbWno5eR0XT5i_oMAghylUBBFOpr27NB3Q,1853
+laketower/templates/tables/import.html,sha256=bQZwRrv84tDBuf0AHJyc7L-PjW-XSoZhMHNDIo6TP4c,2604
+laketower/templates/tables/index.html,sha256=saNdQbJAjMJAzayTk4rA5Mmw_bCXvor2WpghVmoWSAI,2507
+laketower/templates/tables/query.html,sha256=lI_jHcXV-lxMIDyWhNcfFwwJUFLHIICue9usZ5obnx0,3344
+laketower/templates/tables/statistics.html,sha256=yVsu-hNPSQ3-IX-4Z4bukLR4nERvy5PwWENfkW24xyg,1858
+laketower/templates/tables/view.html,sha256=D5jkzzMM6YaFKPquTGnh--7tlNBtF0iztBN0ULvLePs,3947
+laketower-0.6.5.dist-info/METADATA,sha256=HEAXi9UmhTPoKgaVSUm2TrJFsK31dlHwzlgSG7gEoVs,27905
+laketower-0.6.5.dist-info/WHEEL,sha256=qtCwoSJWgHk21S1Kb4ihdzI2rlJ1ZKaIurTj_ngOhyQ,87
+laketower-0.6.5.dist-info/entry_points.txt,sha256=sJpQgRwdeZhRBudNqBTqtHPCE-uLC9YgFXJY2CTEyCk,53
+laketower-0.6.5.dist-info/licenses/LICENSE,sha256=xx0jnfkXJvxRnG63LTGOxlggYnIysveWIZ6H3PNdCrQ,11357
+laketower-0.6.5.dist-info/RECORD,,

laketower-0.6.5.dist-info/entry_points.txt

@@ -0,0 +1,2 @@
+[console_scripts]
+laketower = laketower.__main__:cli

laketower-0.5.1.dist-info/RECORD

@@ -1,22 +0,0 @@
-laketower/__about__.py,sha256=eZ1bOun1DDVV0YLOBW4wj2FP1ajReLjbIrGmzN7ASBw,22
-laketower/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
-laketower/__main__.py,sha256=czKxJKG8OfncnxWmpaOWx7b1JBwFnZNQi7wKSTncB4M,108
-laketower/cli.py,sha256=U4gI12egcOs51wxjmQlU70XhA2QGcowc0AmTYpUKEFE,11962
-laketower/config.py,sha256=NdUDF7lr2hEW9Gujp0OpkOKcDP46ju1y_r0IM4Hrx2M,1100
-laketower/tables.py,sha256=Y-9kf6rwNbAVFgxzm6hAFlEMxGMqHyv7RiveDgMBNf4,4437
-laketower/web.py,sha256=5NMKj26aVz3cKnUAe-3sLDJ_4Ue3u0VXhATrDQ8GVF8,7205
-laketower/static/.gitkeep,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
-laketower/templates/_base.html,sha256=S-8kjAfYBx3Btb4FwzM2qyfkGYrOBHhpvCWR32mCvOw,3729
-laketower/templates/index.html,sha256=dLF2Og0qgzBkvGyVRidRNzTv0u4o97ifOx1jVeig8Kg,59
-laketower/templates/queries/view.html,sha256=2jW08X-PflXmsfy7u9ibX1BF8G5dx9X_n_Eoq3jzizA,1686
-laketower/templates/tables/_macros.html,sha256=fnj_8nBco0iS6mlBmGmfT2PZhI2Y82yP0cm8tRxchpU,965
-laketower/templates/tables/history.html,sha256=yAW0xw9_Uxp0QZYKje6qhcbpeznxI3fb740hfNyILZ8,1740
-laketower/templates/tables/index.html,sha256=oY13l_p8qozlLONanLpga1WhEo4oTP92pRf9sBSuFZI,2394
-laketower/templates/tables/query.html,sha256=YAFnW8Q5abDsbeglFHHZmJfGJXPjI2s4Nxf6gF_-Eg0,1360
-laketower/templates/tables/statistics.html,sha256=rgIOuF2PlHo2jvcYDAnxa5ObNortwyALlrURpM7qxMw,1714
-laketower/templates/tables/view.html,sha256=psfeRKkN19Q3Ko5Sm2570qRhehvuoEBPG89zFU5KQlc,3872
-laketower-0.5.1.dist-info/METADATA,sha256=OrkW0OD9bJ6JnIaPlyEmGJAU8pEWuJrCg9kJRVjwnCY,21353
-laketower-0.5.1.dist-info/WHEEL,sha256=qtCwoSJWgHk21S1Kb4ihdzI2rlJ1ZKaIurTj_ngOhyQ,87
-laketower-0.5.1.dist-info/entry_points.txt,sha256=OL_4klopvyEzasJOFJ-sKu54lv24Jvomni32h1WVUjk,48
-laketower-0.5.1.dist-info/licenses/LICENSE,sha256=xx0jnfkXJvxRnG63LTGOxlggYnIysveWIZ6H3PNdCrQ,11357
-laketower-0.5.1.dist-info/RECORD,,

laketower-0.5.1.dist-info/entry_points.txt

@@ -1,2 +0,0 @@
-[console_scripts]
-laketower = laketower:cli.cli