squirrels 0.5.0b1__tar.gz → 0.5.0b2__tar.gz

{squirrels-0.5.0b1 → squirrels-0.5.0b2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: squirrels
-Version: 0.5.0b1
+Version: 0.5.0b2
 Summary: Squirrels - API Framework for Data Analytics
 Project-URL: Homepage, https://squirrels-analytics.github.io
 Project-URL: Repository, https://github.com/squirrels-analytics/squirrels
@@ -62,8 +62,8 @@ Here are a few of the things that squirrels can do:
 - Configure parameter widgets (types include single-select, multi-select, date, number, etc.) for your datasets (in `parameters.py`).
 - Use Jinja SQL templates (just like dbt!) or python functions (that return a Python dataframe such as polars or pandas) to define dynamic query logic based on parameter selections.
 - Query multiple databases and join the results together in a final view in one API endpoint/dataset!
-- Test your API endpoints with an interactive UI or by a command line that generates rendered sql queries and results (for a given set of parameter selections).
-- Define authentication logic (in `auth.py`) and authorize privacy scope per dataset (in `squirrels.yml`). The user's attributes can even be used in your query logic!
+- Test your API endpoints with Squirrels Studio or by a command line that generates rendered sql queries and results (for a given set of parameter selections).
+- Define User model (in `user.py`) and authorize privacy scope per dataset (in `squirrels.yml`). The user's attributes can even be used in your query logic!
 
 ## License
 
@@ -77,18 +77,18 @@ The sections below describe how to set up your local environment for squirrels d
 
 ### Setup
 
-This project requires python version 3.10 or above to be installed. It also uses the python build tool `poetry`. Information on setting up poetry can be found at: https://python-poetry.org/docs/.
+This project requires python version 3.10 or above to be installed. It also uses the python package manager `uv`. Information on setting up poetry can be found at: https://docs.astral.sh/uv/getting-started/installation/.
 
-Then, to install all dependencies, run:
+Then, to install all dependencies in a virtual environment, run:
 
-```
-poetry install
+```bash
+uv sync
 ```
 
-And activate the virtual environment created by poetry with:
+And activate the virtual environment with:
 
-```
-poetry shell
+```bash
+source .venv/bin/activate
 ```
 
 To confirm that the setup worked, run the following to show the help page for all squirrels CLI commands:
@@ -97,11 +97,9 @@ To confirm that the setup worked, run the following to show the help page for al
 sqrl -h
 ```
 
-You can enter `exit` to exit the virtual environment shell. You can also run `poetry run sqrl -h` to run squirrels commands without activating the virtual environment.
-
 ### Testing
 
-In poetry's virtual environment, run `pytest`.
+Run `uv run pytest`. Or if you have the virtual environment activated, simply run `pytest`.
 
 ### Project Structure
 
@@ -110,7 +108,3 @@ From the root of the git repo, the source code can be found in the `squirrels` f
 To understand what a specific squirrels command is doing, start from the `_command_line.py` file as your entry point.
 
 The library version is maintained in both the `pyproject.toml` and the `squirrels/_version.py` files.
-
-When a user initializes a squirrels project using `sqrl init`, the files are copied from the `squirrels/package_data/base_project` folder. The contents in the `database` subfolder were constructed from the scripts in the `database_elt` folder.
-
-For the Squirrels UI activated by `sqrl run`, the HTML, CSS, and Javascript files can be found in the `static` and `templates` subfolders of `squirrels/package_data`. The CSS and Javascript files are minified and built from the source files in this project: https://github.com/squirrels-analytics/squirrels-testing-ui.

{squirrels-0.5.0b1 → squirrels-0.5.0b2}/README.md

@@ -24,8 +24,8 @@ Here are a few of the things that squirrels can do:
 - Configure parameter widgets (types include single-select, multi-select, date, number, etc.) for your datasets (in `parameters.py`).
 - Use Jinja SQL templates (just like dbt!) or python functions (that return a Python dataframe such as polars or pandas) to define dynamic query logic based on parameter selections.
 - Query multiple databases and join the results together in a final view in one API endpoint/dataset!
-- Test your API endpoints with an interactive UI or by a command line that generates rendered sql queries and results (for a given set of parameter selections).
-- Define authentication logic (in `auth.py`) and authorize privacy scope per dataset (in `squirrels.yml`). The user's attributes can even be used in your query logic!
+- Test your API endpoints with Squirrels Studio or by a command line that generates rendered sql queries and results (for a given set of parameter selections).
+- Define User model (in `user.py`) and authorize privacy scope per dataset (in `squirrels.yml`). The user's attributes can even be used in your query logic!
 
 ## License
 
@@ -39,18 +39,18 @@ The sections below describe how to set up your local environment for squirrels d
 
 ### Setup
 
-This project requires python version 3.10 or above to be installed. It also uses the python build tool `poetry`. Information on setting up poetry can be found at: https://python-poetry.org/docs/.
+This project requires python version 3.10 or above to be installed. It also uses the python package manager `uv`. Information on setting up poetry can be found at: https://docs.astral.sh/uv/getting-started/installation/.
 
-Then, to install all dependencies, run:
+Then, to install all dependencies in a virtual environment, run:
 
-```
-poetry install
+```bash
+uv sync
 ```
 
-And activate the virtual environment created by poetry with:
+And activate the virtual environment with:
 
-```
-poetry shell
+```bash
+source .venv/bin/activate
 ```
 
 To confirm that the setup worked, run the following to show the help page for all squirrels CLI commands:
@@ -59,11 +59,9 @@ To confirm that the setup worked, run the following to show the help page for al
 sqrl -h
 ```
 
-You can enter `exit` to exit the virtual environment shell. You can also run `poetry run sqrl -h` to run squirrels commands without activating the virtual environment.
-
 ### Testing
 
-In poetry's virtual environment, run `pytest`.
+Run `uv run pytest`. Or if you have the virtual environment activated, simply run `pytest`.
 
 ### Project Structure
 
@@ -72,7 +70,3 @@ From the root of the git repo, the source code can be found in the `squirrels` f
 To understand what a specific squirrels command is doing, start from the `_command_line.py` file as your entry point.
 
 The library version is maintained in both the `pyproject.toml` and the `squirrels/_version.py` files.
-
-When a user initializes a squirrels project using `sqrl init`, the files are copied from the `squirrels/package_data/base_project` folder. The contents in the `database` subfolder were constructed from the scripts in the `database_elt` folder.
-
-For the Squirrels UI activated by `sqrl run`, the HTML, CSS, and Javascript files can be found in the `static` and `templates` subfolders of `squirrels/package_data`. The CSS and Javascript files are minified and built from the source files in this project: https://github.com/squirrels-analytics/squirrels-testing-ui.

squirrels-0.5.0b1/squirrels/dateutils.py → squirrels-0.5.0b2/dateutils/__init__.py

@@ -5,8 +5,6 @@ from dateutil.relativedelta import relativedelta
 from abc import ABCMeta, abstractmethod
 from enum import Enum
 
-from . import _utils as u
-
 
 class DayOfWeek(Enum):
     Sunday = 0
@@ -62,7 +60,7 @@ class _DayIdxOfCalendarUnit(DateModifier):
         super().__init__()
         self.idx = idx
         if self.idx == 0:
-            raise u.ConfigurationError(f"For constructors of class names that start with DayIdxOf_, idx cannot be zero")
+            raise ValueError(f"For constructors of class names that start with DayIdxOf_, idx cannot be zero")
         self.incr = self.idx - 1 if self.idx > 0 else self.idx
 
 
@@ -84,7 +82,7 @@ class DayIdxOfMonthsCycle(_DayIdxOfCalendarUnit):
         self._num_months_in_cycle = num_months_in_cycle
         self._first_month_of_cycle = first_month_of_cycle
         if 12 % self._num_months_in_cycle != 0:
-            raise u.ConfigurationError(f"Value X must fit evenly in 12")
+            raise ValueError(f"Value X must fit evenly in 12")
         self.first_month_of_first_cycle = (self._first_month_of_cycle.value - 1) % self._num_months_in_cycle + 1
 
     def modify(self, date: Date) -> Date:
@@ -302,7 +300,7 @@ class DateModPipeline(DateModifier):
         """
         assert isinstance(step, _OffsetUnits)
         if step.offset == 0:
-            raise u.ConfigurationError(f"The length of 'step' must not be zero")
+            raise ValueError(f"The length of 'step' must not be zero")
         
         output: Sequence[Date] = []
         end_date = self.modify(start_date)

{squirrels-0.5.0b1 → squirrels-0.5.0b2}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "squirrels"
-version = "0.5.0b1"
+version = "0.5.0b2"
 description = "Squirrels - API Framework for Data Analytics"
 authors = [{ name = "Tim Huang", email = "tim.yuting@hotmail.com" }]
 requires-python = "~=3.10"
@@ -58,6 +58,12 @@ dev = [
     "tqdm>=4.67.1,<5",
 ]
 
+[tool.hatch.build.targets.sdist]
+include = ["squirrels", "dateutils"]
+
+[tool.hatch.build.targets.wheel]
+include = ["squirrels", "dateutils"]
+
 [tool.uv]
 default-groups = [
     "test",

{squirrels-0.5.0b1 → squirrels-0.5.0b2}/squirrels/_command_line.py

@@ -1,5 +1,5 @@
 from argparse import ArgumentParser, _SubParsersAction
-import sys, asyncio, traceback, io, os, subprocess
+import sys, asyncio, traceback, io, subprocess
 
 sys.path.append('.')
 
@@ -11,10 +11,12 @@ from ._project import SquirrelsProject
 from . import _constants as c, _utils as u
 
 
-def _run_duckdb_cli(project: SquirrelsProject):
+def _run_duckdb_cli(project: SquirrelsProject, ui: bool):
     _, target_init_path = u._read_duckdb_init_sql()
     init_args = f"-init {target_init_path}" if target_init_path else ""
     command = ['duckdb']
+    if ui:
+        command.append('-ui')
     if init_args:
         command.extend(init_args.split())
     command.extend(['-readonly', project._duckdb_venv_path])
@@ -48,21 +50,22 @@ def main():
 
     init_parser = add_subparser(subparsers, c.INIT_CMD, 'Create a new squirrels project')
     
-    init_parser.add_argument('name', nargs='?', type=str, help='The name of the project')
-    init_parser.add_argument('-o', '--overwrite', action='store_true', help="Overwrite files that already exist")
-    init_parser.add_argument('--core', action='store_true', help='Include all core files')
+    init_parser.add_argument('name', nargs='?', type=str, help='The name of the project folder to create. Ignored if --curr-dir is used')
+    init_parser.add_argument('--curr-dir', action='store_true', help='Create the project in the current directory')
+    init_parser.add_argument('--use-defaults', action='store_true', help='Use default values for unspecified options (except project folder name) instead of prompting for input')
     init_parser.add_argument('--connections', type=str, choices=c.CONF_FORMAT_CHOICES, help=f'Configure database connections as yaml (default) or python')
     init_parser.add_argument('--parameters', type=str, choices=c.CONF_FORMAT_CHOICES, help=f'Configure parameters as python (default) or yaml')
     init_parser.add_argument('--build', type=str, choices=c.FILE_TYPE_CHOICES, help='Create build model as sql (default) or python file')
     init_parser.add_argument('--federate', type=str, choices=c.FILE_TYPE_CHOICES, help='Create federated model as sql (default) or python file')
-    init_parser.add_argument('--dashboard', action='store_true', help=f'Include a sample dashboard file')
+    init_parser.add_argument('--dashboard', type=str, choices=['y', 'n'], help=f'Include (y) or exclude (n, default) a sample dashboard file')
+    init_parser.add_argument('--admin-password', type=str, help='The password for the admin user. If --use-defaults is used, then a random password is generated')
 
     def with_file_format_options(parser: ArgumentParser):
         help_text = "Create model as sql (default) or python file"
         parser.add_argument('--format', type=str, choices=c.FILE_TYPE_CHOICES, default=c.SQL_FILE_TYPE, help=help_text)
         return parser
     
-    get_file_help_text = "Get a sample file for the squirrels project. If the file name already exists, it will be prefixed with a timestamp."
+    get_file_help_text = "Get a sample file for the squirrels project. If the file name already exists, it will be suffixed with a timestamp."
     get_file_parser = add_subparser(subparsers, c.GET_FILE_CMD, get_file_help_text)
     get_file_subparsers = get_file_parser.add_subparsers(title='file_name', dest='file_name')
     add_subparser(get_file_subparsers, c.DOTENV_FILE, f'Get sample {c.DOTENV_FILE} and {c.DOTENV_FILE}.example files')
@@ -104,6 +107,7 @@ def main():
     build_parser.add_argument('--stage', type=str, help='If the venv file is in use, stage the duckdb file to replace the venv later')
 
     duckdb_parser = add_subparser(subparsers, c.DUCKDB_CMD, 'Run the duckdb command line tool')
+    duckdb_parser.add_argument('--ui', action='store_true', help='Run the duckdb local UI')
 
     run_parser = add_subparser(subparsers, c.RUN_CMD, 'Run the API server')
     run_parser.add_argument('--build', action='store_true', help='Build the virtual data environment (with duckdb) first before running the API server')
@@ -116,7 +120,7 @@ def main():
     if args.version:
         print(__version__)
     elif args.command == c.INIT_CMD:
-        Initializer(project_name=args.name, overwrite=args.overwrite).init_project(args)
+        Initializer(project_name=args.name, use_curr_dir=args.curr_dir).init_project(args)
     elif args.command == c.GET_FILE_CMD:
         Initializer().get_file(args)
     elif args.command is None:
@@ -131,7 +135,7 @@ def main():
                 asyncio.run(task)
                 print()
             elif args.command == c.DUCKDB_CMD:
-                _run_duckdb_cli(project)
+                _run_duckdb_cli(project, args.ui)
             elif args.command == c.RUN_CMD:
                 if args.build:
                     task = project.build(full_refresh=True)

{squirrels-0.5.0b1 → squirrels-0.5.0b2}/squirrels/_initializer.py

@@ -10,9 +10,9 @@ TMP_FOLDER = "tmp"
 
 
 class Initializer:
-    def __init__(self, *, project_name: Optional[str] = None, overwrite: bool = False):
-        self.project_name = project_name
-        self.overwrite = overwrite
+    def __init__(self, *, project_name: Optional[str] = None, use_curr_dir: bool = False):
+        self.project_name = project_name if not use_curr_dir else None
+        self.use_curr_dir = use_curr_dir
 
     def _path_exists(self, filepath: u.Path) -> bool:
         return os.path.exists(filepath)
@@ -40,8 +40,6 @@ class Initializer:
             if self._files_have_same_content(src_path, filepath2):
                 perform_copy = False
                 extra_msg = "Skipping... file contents is same as source"
-            elif self.overwrite:
-                extra_msg = "Overwriting file..."
             else:
                 filepath2 = self._add_timestamp_to_filename(old_filepath)
                 extra_msg = f'Creating file as "{filepath2}" instead...'
@@ -116,42 +114,63 @@ class Initializer:
         self._copy_file(u.Path(c.DOTENV_FILE + ".example"))
 
     def init_project(self, args):
-        options = ["core", "connections", "parameters", "build", "federate", "dashboard"]
-        _, CONNECTIONS, PARAMETERS, BUILD, FEDERATE, DASHBOARD = options
+        options = ["connections", "parameters", "build", "federate", "dashboard", "admin_password"]
+        CONNECTIONS, PARAMETERS, BUILD, FEDERATE, DASHBOARD, ADMIN_PASSWORD = options
 
         # Add project name prompt if not provided
-        if self.project_name is None:
+        if self.project_name is None and not args.curr_dir:
             questions = [
-                inquirer.Text('project_name', message="What is your project name? (leave blank to create in current directory)")
+                inquirer.Text('project_name', message="What is your project folder name? (leave blank to create in current directory)")
             ]
             answers = inquirer.prompt(questions)
             assert isinstance(answers, dict)
             self.project_name = answers['project_name']
 
         answers = { x: getattr(args, x) for x in options }
-        if not any(answers.values()):
-            questions = [
-                inquirer.List(
-                    CONNECTIONS, message=f"How would you like to configure the database connections?", choices=c.CONF_FORMAT_CHOICES
-                ),
-                inquirer.List(
-                    PARAMETERS, message=f"How would you like to configure the parameters?", choices=c.CONF_FORMAT_CHOICES2
-                ),
-                inquirer.List(
-                    BUILD, message="What's the file format for the build model?", choices=c.FILE_TYPE_CHOICES
-                ),
-                inquirer.List(
-                    FEDERATE, message="What's the file format for the federated model?", choices=c.FILE_TYPE_CHOICES
-                ),
-                inquirer.Confirm(
-                    DASHBOARD, message=f"Do you want to include a dashboard example?", default=False
-                ),
-                inquirer.Password(
-                    "admin_password", message="What's the admin password? (leave blank to generate a random one)"
-                ),
-            ]
-            answers = inquirer.prompt(questions)
-            assert isinstance(answers, dict)
+        if DASHBOARD in answers:
+            answers[DASHBOARD] = (answers[DASHBOARD] == 'y') # convert 'y' or 'n' to boolean
+
+        if not args.use_defaults:
+            questions = []
+            if answers.get(CONNECTIONS) is None:
+                questions.append(
+                    inquirer.List(
+                        CONNECTIONS, message=f"How would you like to configure the database connections?", choices=c.CONF_FORMAT_CHOICES
+                    ),
+                )
+            if answers.get(PARAMETERS) is None:
+                questions.append(
+                    inquirer.List(
+                        PARAMETERS, message=f"How would you like to configure the parameters?", choices=c.CONF_FORMAT_CHOICES2
+                    ),
+                )
+            if answers.get(BUILD) is None:
+                questions.append(
+                    inquirer.List(
+                        BUILD, message="What's the file format for the build model?", choices=c.FILE_TYPE_CHOICES
+                    ),
+                )
+            if answers.get(FEDERATE) is None:
+                questions.append(
+                    inquirer.List(
+                        FEDERATE, message="What's the file format for the federated model?", choices=c.FILE_TYPE_CHOICES
+                    ),
+                )
+            if answers.get(DASHBOARD) is None:
+                questions.append(
+                    inquirer.Confirm(
+                        DASHBOARD, message=f"Do you want to include a dashboard example?", default=False
+                    ),
+                )
+            if answers.get(ADMIN_PASSWORD) is None:
+                questions.append(
+                    inquirer.Password(
+                        "admin_password", message="What's the admin password? (leave blank to generate a random one)"
+                    ),
+                )
+            more_answers = inquirer.prompt(questions)
+            assert isinstance(more_answers, dict)
+            answers.update(more_answers)
         
         def get_answer(key, default):
             """
@@ -177,7 +196,7 @@ class Initializer:
         parameters_use_py = (parameters_format == c.PYTHON_FORMAT)
 
         build_config_file = c.BUILD_FILE_STEM + ".yml"
-        build_format = get_answer(BUILD, c.PYTHON_FILE_TYPE)
+        build_format = get_answer(BUILD, c.SQL_FILE_TYPE)
         if build_format == c.SQL_FILE_TYPE:
             build_file = c.BUILD_FILE_STEM + ".sql"
         elif build_format == c.PYTHON_FILE_TYPE:

{squirrels-0.5.0b1 → squirrels-0.5.0b2}/squirrels/_model_builder.py

@@ -79,6 +79,8 @@ class ModelBuilder:
                     duckdb_stg_path.replace(duckdb_dev_path)
                 elif duckdb_path.exists():
                     shutil.copy(duckdb_path, duckdb_dev_path)
+            else:
+                duckdb_dev_path.unlink(missing_ok=True) # delete any lingering development copy to create a fresh one later
                 
             self._logger.log_activity_time("creating development copy of virtual data environment", start)
         

{squirrels-0.5.0b1 → squirrels-0.5.0b2}/squirrels/_model_configs.py

@@ -47,7 +47,7 @@ class QueryModelConfig(ModelConfig):
 
 
 class BuildModelConfig(QueryModelConfig):
-    materialization: str = Field(default="TABLE", description="The materialization of the model (ignored if Python model which is always a table)")
+    materialization: str = Field(default="VIEW", description="The materialization of the model (ignored if Python model which is always a table)")
 
     def get_sql_for_build(self, model_name: str, select_query: str) -> str:
         if self.materialization.upper() == "TABLE":

{squirrels-0.5.0b1 → squirrels-0.5.0b2}/squirrels/_models.py

@@ -476,15 +476,17 @@ class DbviewModel(QueryModel):
         connection_props = self.conn_set.get_connection(connection_name)
         
         if self.model_config.translate_to_duckdb and isinstance(connection_props, ConnectionProperties):
+            source_func = lambda source_name: "venv." + source_name
             macros = {
-                "source": lambda source_name: "venv." + source_name
+                "source": source_func, "ref": source_func
             }
             compiled_query2 = self._get_compiled_sql_query_str(compiled_query_str, macros)
             compiled_query_str = self._get_duckdb_query(connection_props.dialect, compiled_query2)
             is_duckdb = True
         else:
+            source_func = lambda source_name: self.sources[source_name].get_table()
             macros = {
-                "source": lambda source_name: self.sources[source_name].get_table()
+                "source": source_func, "ref": source_func
             }
             compiled_query_str = self._get_compiled_sql_query_str(compiled_query_str, macros)
             is_duckdb = False

{squirrels-0.5.0b1 → squirrels-0.5.0b2}/squirrels/_utils.py

@@ -20,7 +20,7 @@ polars_dtypes_to_sqrl_dtypes: dict[type[pl.DataType], list[str]] = {
     pl.Int32: ["integer", "int", "int4"],
     pl.Int64: ["bigint", "long", "int8"],
     pl.Float32: ["float", "float4", "real"],
-    pl.Float64: ["double", "float8"],
+    pl.Float64: ["double", "float8", "decimal"], # Note: Polars Decimal type is considered unstable, so we use Float64 for "decimal"
     pl.Boolean: ["boolean", "bool", "logical"],
     pl.Date: ["date"],
     pl.Time: ["time"],

{squirrels-0.5.0b1 → squirrels-0.5.0b2}/squirrels/package_data/base_project/models/builds/build_example.yml

@@ -1,6 +1,8 @@
 description: |
   This is an example of a build model. It adds a new column called "month" to the source table "src_transactions".
 
+materialization: TABLE     # optional - defaults to "VIEW" for SQL models, ignored and always a "TABLE" for Python models
+
 depends_on:                # optional for SQL models - the "ref" macro also adds to this set
   - src_transactions
   - seed_categories

squirrels-0.5.0b1/.cursorignore

@@ -1,2 +0,0 @@
-database_elt/
-poetry.lock

squirrels-0.5.0b1/.github/workflows/python-publish.yml

@@ -1,45 +0,0 @@
-name: Publish Python Package
-
-on:
-  push:
-    tags:
-      - 'v*'
-
-permissions:
-  contents: read
-
-jobs:
-  deploy:
-    runs-on: ubuntu-latest
-    steps:
-      - uses: actions/checkout@v3
-      
-      - name: Set up Python
-        uses: actions/setup-python@v4
-        with:
-          python-version: '3.x'
-          
-      - name: Extract version from tag
-        id: get_version
-        run: echo "VERSION=${GITHUB_REF#refs/tags/v}" >> $GITHUB_OUTPUT
-        
-      - name: Update version in pyproject.toml
-        run: |
-          # Use sed to replace the version in pyproject.toml
-          sed -i "s/^version = \".*\"/version = \"${{ steps.get_version.outputs.VERSION }}\"/" pyproject.toml
-          echo "Updated version in pyproject.toml to ${{ steps.get_version.outputs.VERSION }}"
-          
-      - name: Install build dependencies
-        run: |
-          python -m pip install --upgrade pip
-          pip install build twine
-          
-      - name: Build package
-        run: python -m build
-        
-      - name: Publish to PyPI
-        env:
-          TWINE_USERNAME: __token__
-          TWINE_PASSWORD: ${{ secrets.PYPI_API_TOKEN }}
-        run: |
-          python -m twine upload --username $TWINE_USERNAME --password $TWINE_PASSWORD dist/*

squirrels-0.5.0b1/database_elt/expenses/.gitignore

@@ -1 +0,0 @@
-expenses.db

squirrels-0.5.0b1/database_elt/expenses/create-expenses.py

@@ -1,90 +0,0 @@
-import sqlite3, random, polars as pl, tqdm, numpy as np
-
-def generate_expense_description(vendors=None, items=None, adjectives=None):
-    """
-    Generates a random expense description.
-
-    Args:
-        categories: A list of expense categories (e.g., ["Groceries", "Dining", "Travel", "Utilities"]).
-        vendors: A list of vendor names (e.g., ["Walmart", "Starbucks", "Amazon", "Gas Station"]).
-        items: A list of items purchased (e.g., ["Milk", "Coffee", "Laptop", "Gas"]).
-        adjectives: A list of adjectives to add variety(e.g., ["Monthly", "Quick", "Online", "Unexpected"]).
-
-    Returns:
-        A randomly generated expense description string.
-    """
-
-    if vendors is None:
-        vendors = ["Vendor A", "Vendor B", "Vendor C", "Vendor D", "Vendor E", "Online Store", "Local Shop", "Restaurant"]
-    if items is None:
-        items = ["Item 1", "Item 2", "Item 3", "Item 4", "Item 5", "Service", "Subscription", "Purchase", "Merchandise", "Goods"]
-    if adjectives is None:
-        adjectives = ["Daily", "Monthly", "Quick", "Online", "Unexpected", "Recurring", "Personal", "Business"]
-
-    description_parts = []
-
-    # Choose a category, vendor, and item
-    adjective = random.choice(adjectives)
-    vendor = random.choice(vendors)
-    item = random.choice(items)
-
-    # Build the description
-    description_parts.append(f"{adjective} {item} - {vendor}")
-
-    return " ".join(description_parts).strip()
-
-# Define the number of transactions to generate
-num_transactions = 1_000
-batches = 100
-batch_size = num_transactions // batches
-
-# Generate the data
-descriptions_df = pl.DataFrame({
-    'description': [generate_expense_description() for _ in range(10**4)]
-})
-
-df_list: list[pl.DataFrame] = []
-rng = np.random.default_rng()
-for _ in tqdm.tqdm(range(batches)):
-    df_current = descriptions_df.sample(batch_size, with_replacement=True, shuffle=True)
-    df_current = df_current.with_columns(
-        pl.lit(rng.integers(
-            np.datetime64('2024-01-01').astype(int), 
-            np.datetime64('2025-01-01').astype(int), 
-            size=batch_size
-        ).astype('datetime64[D]')).alias('date'),
-        pl.lit(rng.integers(0, 14, size=batch_size)).alias('subcategory_id'),
-        pl.lit(rng.exponential(30, size=batch_size).round(2)).alias('amount'),
-    )
-    df_list.append(df_current)
-
-df = pl.concat(df_list).select('date', 'subcategory_id', 'amount', 'description')
-df = df.sort('date').with_row_index(name='id', offset=1)
-
-# Connect to SQLite database
-conn = sqlite3.connect('expenses.db')
-
-try:
-    # Create the expenses table
-    conn.execute("DROP TABLE IF EXISTS expenses")
-    conn.execute('''
-    CREATE TABLE IF NOT EXISTS expenses (
-        id INTEGER PRIMARY KEY,
-        date DATE,
-        subcategory_id INTEGER,
-        amount DECIMAL(10,2),
-        description TEXT
-    )
-    ''')
-
-    # Convert DataFrame to records and insert into database
-    records = df.to_numpy().tolist()
-    conn.executemany(
-        'INSERT INTO expenses (id, date, subcategory_id, amount, description) VALUES (?, ?, ?, ?, ?)',
-        records
-    )
-
-    # Commit changes and close connection
-    conn.commit()
-finally:
-    conn.close()

squirrels-0.5.0b1/database_elt/expenses/create-lookups.py

@@ -1,55 +0,0 @@
-import sqlite3
-import csv
-from io import StringIO
-
-category_id_mapping = """
-"category_id","subcategory_id"
-0,0
-0,1
-1,2
-2,3
-3,4
-1,5
-2,6
-1,7
-4,8
-4,9
-2,10
-3,11
-2,12
-4,13
-"""
-
-# Connect to the SQLite database
-conn = sqlite3.connect('expenses.db')
-try:
-    cursor = conn.cursor()
-
-    # Create the category_mapping table
-    cursor.execute('''
-    DROP TABLE IF EXISTS category_mapping
-    ''')
-    cursor.execute('''
-    CREATE TABLE category_mapping (
-        category_id INTEGER,
-        subcategory_id INTEGER,
-        PRIMARY KEY (subcategory_id)
-    )
-    ''')
-
-    # Parse the CSV string and insert data
-    csv_file = StringIO(category_id_mapping.strip())
-    csv_reader = csv.DictReader(csv_file)
-
-    # Insert the data
-    for row in csv_reader:
-        cursor.execute('''
-        INSERT OR REPLACE INTO category_mapping (category_id, subcategory_id)
-        VALUES (?, ?)
-        ''', (int(row['category_id']), int(row['subcategory_id'])))
-
-    # Commit the changes and close the connection
-    conn.commit()
-
-finally:
-    conn.close()

squirrels-0.5.0b1/database_elt/seattle_weather/create_db.py

@@ -1,13 +0,0 @@
-import pandas as pd
-import sqlite3, os
-
-os.chdir(os.path.dirname(__file__))
-
-df = pd.read_csv('seattle-weather.csv')
-
-# Connect to SQLite database
-conn = sqlite3.connect('weather.db')
-try:
-    df.to_sql('weather', conn, index=False, if_exists='replace')
-finally:
-    conn.close()