fullscript.spellbook 3.0.0__tar.gz → 3.0.2__tar.gz

{fullscript_spellbook-3.0.0 → fullscript_spellbook-3.0.2}/AGENTS.md

@@ -58,6 +58,15 @@ pattern instead.
 
 ## Traps that have cost time before
 
+**Validate the database being used, not the whole config.** The entire `databases` section
+is re-parsed on every lookup, so any check that runs during parsing applies to databases the
+caller never asked for. Release 3.0.0 raised on the first unset `${VAR}` anywhere in the
+section, which meant an unrelated Snowflake entry with no key passphrase blocked every MySQL
+query. `parse_env_variables` now substitutes permissively and records unresolved names under
+`_unresolved_env`; `db_magic._validate_db_config` turns those into an error for the selected
+database only, and only for settings in `REQUIRED_FIELDS`. Settings that are legitimately
+empty - a Snowflake passphrase, `ssl_ca`, `port`, `schema` - must stay out of that list.
+
 **Config file precedence.** `utils.find_config_file()` prefers `spellbook_config.yaml` in
 the *current working directory* over the one in your home directory. Running anything from
 this repository picks up the placeholder example config, not your real one. If a connection

{fullscript_spellbook-3.0.0 → fullscript_spellbook-3.0.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: fullscript.spellbook
-Version: 3.0.0
+Version: 3.0.2
 Summary: A simple package to make your life easier.
 Author-email: Meinhard Mare <meinhard.mare@fullscript.com>, Thomas Edwards <thomas.edwards@fullscript.com>, Gui Chagas <gui.chagas@fullscript.com>, David King <david.king@fullscript.com>, Karsten Melchers <karsten.melchers@fullscript.com>
 License: MIT
@@ -177,25 +177,34 @@ An example of the `spellbook_config.yaml` file is shown below:
 ```yaml
 databases:
   # Snowflake Database Connection
-  - name: alias_of_your_connection
+  - name: snowflake_example
     type: snowflake
-    account: your_account
-    user: snowflake_user_name # Optional for private key auth
-    password: snowflake_password # Optional, used if private key is not present
-    private_key: snowflake_pk # Base64-encoded private key
-    private_key_passphrase: snowflake_pk_phrase # Optional
+    account: ${SNOWFLAKE_ACCOUNT}
+    user: ${SNOWFLAKE_USER}
+    password: ${SNOWFLAKE_PASSWORD} # Optional when a private key is given
+    private_key: ${SNOW_PK} # Base64-encoded; used in preference to the password
+    private_key_passphrase: ${SNOW_PK_PASSPHRASE} # Optional
     role: snowflake_role
     warehouse: snowflake_warehouse
     database: snowflake_database
-    schema: snowflake_schema
+    schema: snowflake_schema # Optional, defaults to 'public'
+
+  # MySQL Database Connection using a stored password
+  - name: mysql_example
+    type: mysql
+    host: your_host
+    port: 3306
+    user: mysql_user_name
+    password: ${MYSQL_PASSWORD}
+    database: mysql_database
 
   # MySQL Database Connection using AWS RDS IAM authentication (no stored password).
   # Fullscript internal only: requires the `rx` CLI. See documentation/db_magic.md.
-  - name: alias_of_your_connection
+  # No `user` is needed - the IAM token names the user it was issued for.
+  - name: mysql_iam_example
     type: mysql
     host: your_host.rds.amazonaws.com
     port: 3306
-    user: ${DB_USER}
     iam_auth: true
     database: ${MYSQL_DATABASE}
 

{fullscript_spellbook-3.0.0 → fullscript_spellbook-3.0.2}/README.md

@@ -54,25 +54,34 @@ An example of the `spellbook_config.yaml` file is shown below:
 ```yaml
 databases:
   # Snowflake Database Connection
-  - name: alias_of_your_connection
+  - name: snowflake_example
     type: snowflake
-    account: your_account
-    user: snowflake_user_name # Optional for private key auth
-    password: snowflake_password # Optional, used if private key is not present
-    private_key: snowflake_pk # Base64-encoded private key
-    private_key_passphrase: snowflake_pk_phrase # Optional
+    account: ${SNOWFLAKE_ACCOUNT}
+    user: ${SNOWFLAKE_USER}
+    password: ${SNOWFLAKE_PASSWORD} # Optional when a private key is given
+    private_key: ${SNOW_PK} # Base64-encoded; used in preference to the password
+    private_key_passphrase: ${SNOW_PK_PASSPHRASE} # Optional
     role: snowflake_role
     warehouse: snowflake_warehouse
     database: snowflake_database
-    schema: snowflake_schema
+    schema: snowflake_schema # Optional, defaults to 'public'
+
+  # MySQL Database Connection using a stored password
+  - name: mysql_example
+    type: mysql
+    host: your_host
+    port: 3306
+    user: mysql_user_name
+    password: ${MYSQL_PASSWORD}
+    database: mysql_database
 
   # MySQL Database Connection using AWS RDS IAM authentication (no stored password).
   # Fullscript internal only: requires the `rx` CLI. See documentation/db_magic.md.
-  - name: alias_of_your_connection
+  # No `user` is needed - the IAM token names the user it was issued for.
+  - name: mysql_iam_example
     type: mysql
     host: your_host.rds.amazonaws.com
     port: 3306
-    user: ${DB_USER}
     iam_auth: true
     database: ${MYSQL_DATABASE}
 

{fullscript_spellbook-3.0.0 → fullscript_spellbook-3.0.2}/documentation/db_magic.md

@@ -81,16 +81,18 @@ tokens expire after about 15 minutes:
   type: mysql
   host: ${MYSQL_HOST} # e.g. your-instance.abc123.us-east-1.rds.amazonaws.com
   port: 3306
-  user: ${DB_USER}
   iam_auth: true
   rds_db_name: your_rds_instance_name # Optional, defaults to the value of `name`
   database: ${MYSQL_DATABASE}
 ```
 
+No `user` is needed: the IAM token names the user it was issued for, and Spellbook reads it
+from the token. Set `user` only if you want to pin it, in which case it must match the token.
+
 | Field | Required | Notes |
 | --- | --- | --- |
 | `iam_auth` | No | `true` switches the database to IAM tokens. Omitting it keeps password authentication. |
-| `user` | Yes | Your IAM database user. |
+| `user` | No | Read from the token's `DBUser`. Set it only to pin a specific user; a value that disagrees with the token is reported as a config error. |
 | `rds_db_name` | No | The remote database as named by `rx rds list`. Defaults to `name`, so set it whenever your config alias differs from the RDS name. |
 | `ssl_ca` | No | Not needed in normal use; see below. |
 | `password` | No | Ignored when `iam_auth` is true. |

{fullscript_spellbook-3.0.0 → fullscript_spellbook-3.0.2}/fullscript.spellbook.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: fullscript.spellbook
-Version: 3.0.0
+Version: 3.0.2
 Summary: A simple package to make your life easier.
 Author-email: Meinhard Mare <meinhard.mare@fullscript.com>, Thomas Edwards <thomas.edwards@fullscript.com>, Gui Chagas <gui.chagas@fullscript.com>, David King <david.king@fullscript.com>, Karsten Melchers <karsten.melchers@fullscript.com>
 License: MIT
@@ -177,25 +177,34 @@ An example of the `spellbook_config.yaml` file is shown below:
 ```yaml
 databases:
   # Snowflake Database Connection
-  - name: alias_of_your_connection
+  - name: snowflake_example
     type: snowflake
-    account: your_account
-    user: snowflake_user_name # Optional for private key auth
-    password: snowflake_password # Optional, used if private key is not present
-    private_key: snowflake_pk # Base64-encoded private key
-    private_key_passphrase: snowflake_pk_phrase # Optional
+    account: ${SNOWFLAKE_ACCOUNT}
+    user: ${SNOWFLAKE_USER}
+    password: ${SNOWFLAKE_PASSWORD} # Optional when a private key is given
+    private_key: ${SNOW_PK} # Base64-encoded; used in preference to the password
+    private_key_passphrase: ${SNOW_PK_PASSPHRASE} # Optional
     role: snowflake_role
     warehouse: snowflake_warehouse
     database: snowflake_database
-    schema: snowflake_schema
+    schema: snowflake_schema # Optional, defaults to 'public'
+
+  # MySQL Database Connection using a stored password
+  - name: mysql_example
+    type: mysql
+    host: your_host
+    port: 3306
+    user: mysql_user_name
+    password: ${MYSQL_PASSWORD}
+    database: mysql_database
 
   # MySQL Database Connection using AWS RDS IAM authentication (no stored password).
   # Fullscript internal only: requires the `rx` CLI. See documentation/db_magic.md.
-  - name: alias_of_your_connection
+  # No `user` is needed - the IAM token names the user it was issued for.
+  - name: mysql_iam_example
     type: mysql
     host: your_host.rds.amazonaws.com
     port: 3306
-    user: ${DB_USER}
     iam_auth: true
     database: ${MYSQL_DATABASE}
 

{fullscript_spellbook-3.0.0 → fullscript_spellbook-3.0.2}/fullscript.spellbook.egg-info/scm_version.json

@@ -1,7 +1,7 @@
 {
-  "tag": "3.0.0",
+  "tag": "3.0.2",
   "distance": 0,
-  "node": "g5c9c10546f95ea303f36c2cc2e273cf241b92215",
+  "node": "gdb663ba3a57f8425421f12ce4d2b484ea72f5022",
   "dirty": false,
   "branch": "HEAD",
   "node_date": "2026-10-22"

{fullscript_spellbook-3.0.0 → fullscript_spellbook-3.0.2}/spellbook/db_magic.py

@@ -4,7 +4,7 @@ import os
 import re
 import subprocess
 from typing import Any
-from urllib.parse import quote_plus
+from urllib.parse import parse_qs, quote_plus
 
 import pandas as pd
 from cryptography.hazmat.backends import default_backend
@@ -20,7 +20,13 @@ from spellbook.exceptions import (
     DatabaseNotFoundError,
     QueryError,
 )
-from spellbook.utils import load_and_parse_config, load_sql_file, show_wizard
+from spellbook.utils import (
+    CONFIG_FILENAME,
+    describe_unresolved,
+    load_and_parse_config,
+    load_sql_file,
+    show_wizard,
+)
 
 logger = logging.getLogger(__name__)
 
@@ -40,6 +46,50 @@ _IAM_CURSOR_CLASS = None
 
 SUPPORTED_DB_TYPES = ('snowflake', 'mysql')
 
+# Settings a connection genuinely cannot be made without, checked only for the database
+# being used. Anything absent from here is optional and may resolve to an empty value:
+# `private_key_passphrase`, `schema`, `port`, `ssl_ca` and `rds_db_name` all have defaults
+# or meaningful empty behaviour. See required_fields() for the per-database adjustments.
+REQUIRED_FIELDS = {
+    'mysql': ('host', 'user', 'database'),
+    'snowflake': ('account', 'user', 'role', 'warehouse', 'database'),
+}
+
+
+def required_fields(db_config: dict[str, Any]) -> tuple[str, ...]:
+    """
+    Work out which settings this particular database cannot do without.
+
+    MySQL needs a `password` unless it authenticates with an IAM token, and conversely an
+    IAM database does not need a `user`, because the token says who it was issued to.
+
+    :param db_config: The configuration block for the database.
+    :return: The names of the settings that must be present and non-empty.
+    """
+    fields = REQUIRED_FIELDS.get(db_config.get('type'), ())
+
+    if db_config.get('type') == 'mysql':
+        if db_config.get('iam_auth'):
+            return tuple(field for field in fields if field != 'user')
+        return fields + ('password',)
+
+    return fields
+
+
+def db_user_from_token(token: str) -> str | None:
+    """
+    Read the database user out of an AWS RDS IAM token.
+
+    The token is a presigned URL and carries the user it was issued for as its `DBUser`
+    parameter, so the user does not have to be repeated in the configuration file.
+
+    :param token: The token returned by `rx rds password`.
+    :return: The database user, or None if the token does not name one.
+    """
+    _, _, query = (token or '').partition('?')
+    users = parse_qs(query).get('DBUser') or []
+    return users[0] or None if users else None
+
 
 def _strip_ansi(text_value: str | None) -> str:
     """
@@ -51,13 +101,40 @@ def _strip_ansi(text_value: str | None) -> str:
     return _ANSI_ESCAPE.sub('', text_value or '')
 
 
+def _validate_db_config(db_name: str, db_config: dict[str, Any]) -> dict[str, Any]:
+    """
+    Check that the settings this database cannot work without are actually filled in.
+
+    Only the database being used is checked. Settings that are legitimately optional, such as
+    a Snowflake private key passphrase or an `ssl_ca` bundle, are allowed to stay empty.
+
+    :param db_name: The name of the database, for the error message.
+    :param db_config: The configuration block for that database.
+    :return: The same configuration block.
+    :raises ConfigError: If a required setting is missing or resolved to an empty value.
+    """
+    unresolved = db_config.pop('_unresolved_env', {})
+
+    for key in required_fields(db_config):
+        if db_config.get(key):
+            continue
+        if key in unresolved:
+            raise ConfigError(describe_unresolved(db_name, key, unresolved[key]))
+        raise ConfigError(
+            f"Database '{db_name}' is missing the required '{key}' setting. "
+            f"Add it to {CONFIG_FILENAME}."
+        )
+
+    return db_config
+
+
 def get_db_config(db_name: str) -> dict[str, Any]:
     """
     Look up a single database block from the configuration file.
 
     :param db_name: The name of the database as defined in the config file.
     :return: The configuration for that database.
-    :raises ConfigError: If the configuration file cannot be read.
+    :raises ConfigError: If the config cannot be read, or this database is incomplete.
     :raises DatabaseNotFoundError: If no database matches the given name.
     """
     config = load_and_parse_config(configurations='databases')
@@ -65,7 +142,7 @@ def get_db_config(db_name: str) -> dict[str, Any]:
     databases = config.get('databases') or []
     for db in databases:
         if db.get('name') == db_name:
-            return db
+            return _validate_db_config(db_name, db)
 
     available = ', '.join(sorted(str(db.get('name')) for db in databases)) or '<none>'
     raise DatabaseNotFoundError(
@@ -224,11 +301,30 @@ def mysql_connect(db_name: str) -> Connection:
         # RDS instance names rarely match the alias used in the config file, so allow an
         # override and fall back to the alias when the two happen to line up.
         token = get_rds_iam_token(db_config.get('rds_db_name', db_config['name']))
+
+        # The token names the user it was issued for, so `user` is optional in the config.
+        token_user = db_user_from_token(token)
+        user = db_config.get('user') or token_user
+        if not user:
+            raise ConfigError(
+                f"Database '{db_name}' has no 'user', and the token from 'rx rds password' "
+                f"did not name one either. Add a 'user' to {CONFIG_FILENAME}."
+            )
+        if token_user and db_config.get('user') and db_config['user'] != token_user:
+            # The token is signed for one user, so connecting as another is rejected by RDS
+            # with a bare access-denied. Say what is actually wrong instead.
+            raise ConfigError(
+                f"Database '{db_name}' is configured for user "
+                f"{db_config['user']!r}, but 'rx rds password' issued a token for "
+                f"{token_user!r}. Remove 'user' from {CONFIG_FILENAME} to use the token's "
+                "own user, or correct it to match."
+            )
+
         # IAM tokens are URL shaped, so they have to be escaped inside the connection
         # string. AWS also requires TLS, and only pymysql completes the token handshake.
         ssl_ca = db_config.get('ssl_ca')
         connection_string = (
-            f"mysql+pymysql://{quote_plus(db_config['user'])}:{quote_plus(token)}"
+            f"mysql+pymysql://{quote_plus(user)}:{quote_plus(token)}"
             f"@{host}:{port}/{database}"
         )
         # A CA bundle gets the certificate verified too; without one we still insist on

{fullscript_spellbook-3.0.0 → fullscript_spellbook-3.0.2}/spellbook/utils.py

@@ -69,13 +69,16 @@ def parse_env_variables(config: dict[str, Any], configurations: str) -> dict[str
     """
     Replace ${VAR} placeholders in a configuration block with environment variable values.
 
-    An unset variable is an error rather than an empty string, because an empty username or
-    host produces a confusing driver-level failure much later on.
+    An unset variable becomes an empty string here rather than an error. The whole section is
+    parsed on every lookup, so refusing to parse would mean an unrelated database you never
+    asked for could stop you connecting to the one you did. Unresolved names are recorded
+    under `_unresolved_env` so that whoever actually uses the block can report them, and so
+    that genuinely optional settings - a Snowflake key with no passphrase, say - still work.
 
     :param config: The configuration dictionary.
     :param configurations: The key of the configuration block to parse.
     :return: The configuration with placeholders substituted.
-    :raises ConfigError: If the block is missing, or a referenced variable is not set.
+    :raises ConfigError: If the section is missing.
     """
     if configurations not in config:
         raise ConfigError(
@@ -85,7 +88,9 @@ def parse_env_variables(config: dict[str, Any], configurations: str) -> dict[str
 
     for block in config[configurations]:
         name = block.get('name', '<unnamed>')
-        for key, value in block.items():
+        unresolved: dict[str, str] = {}
+
+        for key, value in list(block.items()):
             if not isinstance(value, str):
                 continue
 
@@ -102,18 +107,33 @@ def parse_env_variables(config: dict[str, Any], configurations: str) -> dict[str
                 continue
 
             for match in matches:
-                env_value = os.getenv(match)
-                if env_value is None or env_value == '':
-                    raise ConfigError(
-                        f"Environment variable '{match}' is referenced by '{key}' on "
-                        f"'{name}' but is not set. Export it, or replace ${{{match}}} in "
-                        f"{CONFIG_FILENAME} with a literal value."
-                    )
+                env_value = os.getenv(match, '')
+                if env_value == '':
+                    unresolved[key] = match
                 block[key] = block[key].replace(f'${{{match}}}', env_value)
 
+        if unresolved:
+            block['_unresolved_env'] = unresolved
+
     return config
 
 
+def describe_unresolved(name: str, key: str, variable: str) -> str:
+    """
+    Build the error message for a required setting whose environment variable is not set.
+
+    :param name: The name of the database the setting belongs to.
+    :param key: The configuration key that is empty.
+    :param variable: The environment variable that was referenced but not set.
+    :return: The message to raise.
+    """
+    return (
+        f"Environment variable '{variable}' is referenced by '{key}' on '{name}' but is "
+        f"not set. Export it, or replace ${{{variable}}} in {CONFIG_FILENAME} with a "
+        "literal value."
+    )
+
+
 def find_config_file(config_file_path: str | None = None) -> str:
     """
     Work out which configuration file will be used.

fullscript_spellbook-3.0.2/spellbook_config.yaml

@@ -0,0 +1,55 @@
+databases:
+  # Snowflake connection, with values written straight into this file.
+  # Spellbook warns about hardcoded credentials, so prefer the next example.
+  - name: snowflake_example
+    type: snowflake
+    account: your_account
+    user: snowflake_user_name
+    password: snowflake_password # Optional when a private key is given
+    private_key: snowflake_pk # Base64-encoded; used in preference to the password
+    private_key_passphrase: snowflake_pk_phrase # Optional, omit if the key has none
+    role: snowflake_role
+    warehouse: snowflake_warehouse
+    database: snowflake_database
+    schema: snowflake_schema # Optional, defaults to 'public'
+
+  # The same connection, reading its sensitive values from environment variables.
+  - name: snowflake_env_example
+    type: snowflake
+    account: ${SNOWFLAKE_ACCOUNT}
+    user: ${SNOWFLAKE_USER}
+    password: ${SNOWFLAKE_PASSWORD}
+    private_key: ${SNOW_PK}
+    private_key_passphrase: ${SNOW_PK_PASSPHRASE} # Optional
+    role: snowflake_role
+    warehouse: snowflake_warehouse
+    database: snowflake_database
+    schema: snowflake_schema
+
+  # MySQL connection using a stored password.
+  - name: mysql_example
+    type: mysql
+    host: your_host
+    port: 3306 # Optional, defaults to 3306
+    user: mysql_user_name
+    password: ${MYSQL_PASSWORD}
+    database: mysql_database
+
+  # MySQL connection using AWS RDS IAM authentication, so no password is stored.
+  # Spellbook fetches a temporary token on every connection, because they expire
+  # after about 15 minutes. Requires the `rx` CLI, which is Fullscript internal tooling.
+  - name: mysql_iam_example
+    type: mysql
+    host: ${MYSQL_HOST} # e.g. your-instance.abc123.us-east-1.rds.amazonaws.com
+    port: 3306
+    iam_auth: true # Use an AWS RDS IAM token instead of a password
+    rds_db_name: your_rds_instance_name # Optional, as named by `rx rds list`; defaults to `name`
+    database: ${MYSQL_DATABASE}
+
+google_accounts:
+  # Google Sheets Service Account 1
+  - name: google_sheets_service_account_1
+    service_account: "your_base64_encoded_service_account_credentials_here"  # Add your Base64-encoded service account JSON here
+  # Google Sheets Service Account 2
+  - name: google_sheets_service_account_2
+    service_account: "your_base64_encoded_service_account_credentials_here"

{fullscript_spellbook-3.0.0 → fullscript_spellbook-3.0.2}/tests/test_db_magic.py

@@ -211,6 +211,94 @@ def test_iam_auth_fetches_a_fresh_token_per_connection(monkeypatch, config, engi
     assert 'token-two' in engines[1][0]
 
 
+# --- The IAM token names its own user ----------------------------------------------------
+# An RDS IAM token is a presigned URL carrying DBUser, so `user` need not be configured.
+
+@pytest.mark.parametrize('token, expected', [
+    (FAKE_TOKEN, 'iam_user'),
+    ('host:3306/?Action=connect&DBUser=first.last&X-Amz-Signature=x', 'first.last'),
+    ('host:3306/?DBUser=needs%40escaping', 'needs@escaping'),
+    ('host:3306/?Action=connect&X-Amz-Signature=x', None),   # no DBUser at all
+    ('host:3306/?DBUser=', None),                             # present but empty
+    ('not-a-url-at-all', None),
+    ('', None),
+])
+def test_db_user_is_read_from_the_token(token, expected):
+    assert db_magic.db_user_from_token(token) == expected
+
+
+def test_user_is_derived_from_the_token_when_not_configured(monkeypatch, config, engines):
+    without_user = {key: value for key, value in IAM_DB.items() if key != 'user'}
+    config(without_user)
+    monkeypatch.setattr(subprocess, 'run', fake_rx(stdout=FAKE_TOKEN))
+
+    db_magic.mysql_connect('reporting_db')
+
+    # FAKE_TOKEN carries DBUser=iam_user
+    assert engines[0][0].startswith('mysql+pymysql://iam_user:')
+
+
+def test_a_configured_user_is_still_honoured(monkeypatch, config, engines):
+    config(IAM_DB)
+    monkeypatch.setattr(subprocess, 'run', fake_rx(stdout=FAKE_TOKEN))
+
+    db_magic.mysql_connect('reporting_db')
+
+    assert engines[0][0].startswith('mysql+pymysql://iam_user:')
+
+
+def test_a_user_that_disagrees_with_the_token_is_reported_clearly(
+    monkeypatch, config, engines,
+):
+    """RDS rejects a mismatch with a bare access-denied, so catch it with a real message."""
+    config({**IAM_DB, 'user': 'someone.else'})
+    monkeypatch.setattr(subprocess, 'run', fake_rx(stdout=FAKE_TOKEN))
+
+    with pytest.raises(ConfigError) as error:
+        db_magic.mysql_connect('reporting_db')
+
+    message = str(error.value)
+    assert 'someone.else' in message
+    assert 'iam_user' in message
+    assert engines == [], 'no connection should be attempted'
+
+
+def test_a_token_without_a_user_and_no_config_user_raises(monkeypatch, config, engines):
+    without_user = {key: value for key, value in IAM_DB.items() if key != 'user'}
+    config(without_user)
+    monkeypatch.setattr(subprocess, 'run', fake_rx(
+        stdout='host:3306/?Action=connect&X-Amz-Signature=x',
+    ))
+
+    with pytest.raises(ConfigError, match="did not name one either"):
+        db_magic.mysql_connect('reporting_db')
+
+
+def test_user_is_not_required_in_config_for_iam_databases(config_file):
+    config_file({'name': 'db', 'type': 'mysql', 'host': 'h',
+                 'database': 'd', 'iam_auth': True})
+
+    assert db_magic.get_db_config('db').get('user') is None
+
+
+def test_user_is_still_required_for_password_databases(config_file):
+    config_file({'name': 'db', 'type': 'mysql', 'host': 'h',
+                 'database': 'd', 'password': 'p'})
+
+    with pytest.raises(ConfigError, match="required 'user'"):
+        db_magic.get_db_config('db')
+
+
+@pytest.mark.parametrize('db_config, expected', [
+    ({'type': 'mysql', 'iam_auth': True}, ('host', 'database')),
+    ({'type': 'mysql'}, ('host', 'user', 'database', 'password')),
+    ({'type': 'snowflake'}, ('account', 'user', 'role', 'warehouse', 'database')),
+    ({'type': 'postgres'}, ()),
+])
+def test_required_fields_per_database_type(db_config, expected):
+    assert db_magic.required_fields(db_config) == expected
+
+
 # --- Literal '%' must survive on the pymysql path ----------------------------------------
 # pymysql substitutes whenever args is not None, so the empty parameter set pandas passes
 # would turn `like '%x%'` into a format string. mysql-connector guards on `if params:`,
@@ -346,6 +434,101 @@ def test_missing_required_field_raises_config_error(config):
         db_magic.mysql_connect('legacy_mysql')
 
 
+# --- Required settings are validated per database, not across the whole config -----------
+# Regression: validation ran over every block while parsing, so an unrelated Snowflake
+# entry with an unset ${SNOW_PK_PASSPHRASE} made every MySQL query fail.
+
+@pytest.fixture
+def config_file(tmp_path, monkeypatch):
+    """Write a real config file and make Spellbook load it."""
+    import yaml
+
+    def write(*databases):
+        path = tmp_path / 'spellbook_config.yaml'
+        path.write_text(yaml.safe_dump({'databases': list(databases)}))
+        monkeypatch.setattr(
+            db_magic, 'load_and_parse_config',
+            lambda **kw: __import__('spellbook.utils', fromlist=['x'])
+            .load_and_parse_config(config_file_path=str(path), **kw),
+        )
+    return write
+
+
+def test_an_unrelated_database_does_not_block_the_one_you_asked_for(
+    monkeypatch, config_file,
+):
+    monkeypatch.delenv('SNOW_PK_PASSPHRASE', raising=False)
+    config_file(
+        {'name': 'snow_s', 'type': 'snowflake', 'account': 'a', 'user': 'u', 'role': 'r',
+         'warehouse': 'w', 'database': 'd',
+         'private_key_passphrase': '${SNOW_PK_PASSPHRASE}'},
+        IAM_DB,
+    )
+
+    db = db_magic.get_db_config('reporting_db')
+
+    assert db['user'] == 'iam_user'
+    assert '_unresolved_env' not in db, 'the internal marker must not leak to callers'
+
+
+def test_an_optional_unset_setting_does_not_block_its_own_database(monkeypatch, config_file):
+    """A Snowflake key with no passphrase is a valid setup, so it must not raise."""
+    monkeypatch.delenv('SNOW_PK_PASSPHRASE', raising=False)
+    config_file({
+        'name': 'snow_s', 'type': 'snowflake', 'account': 'a', 'user': 'u', 'role': 'r',
+        'warehouse': 'w', 'database': 'd', 'private_key_passphrase': '${SNOW_PK_PASSPHRASE}',
+    })
+
+    db = db_magic.get_db_config('snow_s')
+
+    assert db['private_key_passphrase'] == ''
+
+
+def test_a_required_setting_with_an_unset_variable_raises_and_names_it(
+    monkeypatch, config_file,
+):
+    monkeypatch.delenv('DB_USER_NOT_SET', raising=False)
+    config_file({
+        'name': 'broken_db', 'type': 'mysql', 'host': 'h',
+        'user': '${DB_USER_NOT_SET}', 'password': 'p', 'database': 'd',
+    })
+
+    with pytest.raises(ConfigError) as error:
+        db_magic.get_db_config('broken_db')
+
+    message = str(error.value)
+    assert 'DB_USER_NOT_SET' in message
+    assert 'broken_db' in message
+    assert 'user' in message
+
+
+@pytest.mark.parametrize('missing', ['host', 'user', 'database'])
+def test_a_missing_required_mysql_setting_raises(config_file, missing):
+    config_file({
+        key: value for key, value in
+        {'name': 'db', 'type': 'mysql', 'host': 'h', 'user': 'u',
+         'password': 'p', 'database': 'd'}.items()
+        if key != missing
+    })
+
+    with pytest.raises(ConfigError, match=f"required '{missing}'"):
+        db_magic.get_db_config('db')
+
+
+def test_password_is_required_without_iam_auth(config_file):
+    config_file({'name': 'db', 'type': 'mysql', 'host': 'h', 'user': 'u', 'database': 'd'})
+
+    with pytest.raises(ConfigError, match="required 'password'"):
+        db_magic.get_db_config('db')
+
+
+def test_password_is_not_required_with_iam_auth(config_file):
+    config_file({'name': 'db', 'type': 'mysql', 'host': 'h', 'user': 'u',
+                 'database': 'd', 'iam_auth': True})
+
+    assert db_magic.get_db_config('db')['iam_auth'] is True
+
+
 def test_every_spellbook_error_shares_one_base_class():
     for error in (
         ConfigError, DatabaseNotFoundError, AuthenticationError,

{fullscript_spellbook-3.0.0 → fullscript_spellbook-3.0.2}/tests/test_utils.py

@@ -38,31 +38,64 @@ def forget_hardcoded_warnings():
     utils._WARNED_HARDCODED.clear()
 
 
-# --- Unresolved environment variables are an error ---------------------------------------
-# Previously ${VAR} for an unset variable became '', which surfaced much later as a
-# confusing driver error, e.g. an empty username being rejected by the server.
-
-def test_unset_environment_variable_raises(monkeypatch, config_file):
-    monkeypatch.delenv('DB_USER', raising=False)
+# --- Unresolved environment variables are recorded, not raised ---------------------------
+# Parsing must stay permissive: the whole section is re-parsed on every lookup, so raising
+# here would let an unrelated database stop you connecting to the one you asked for.
+# db_magic.get_db_config turns these records into an error for the database actually used.
+
+@pytest.mark.parametrize('unset_value', [None, ''])
+def test_unset_environment_variable_is_recorded_not_raised(
+    monkeypatch, config_file, unset_value,
+):
     monkeypatch.setenv('MYSQL_HOST', 'db.example.com')
+    if unset_value is None:
+        monkeypatch.delenv('DB_USER', raising=False)
+    else:
+        monkeypatch.setenv('DB_USER', unset_value)
     path = config_file(CONFIG)
 
-    with pytest.raises(ConfigError) as error:
-        utils.load_and_parse_config(config_file_path=path, configurations='databases')
+    config = utils.load_and_parse_config(config_file_path=path, configurations='databases')
 
-    message = str(error.value)
-    assert 'DB_USER' in message
-    assert 'reporting_db' in message, 'the error should name the database involved'
-    assert 'user' in message
+    db = config['databases'][0]
+    assert db['user'] == '', 'the placeholder is substituted away'
+    assert db['_unresolved_env'] == {'user': 'DB_USER'}
+    assert db['host'] == 'db.example.com', 'resolved values are unaffected'
 
 
-def test_empty_environment_variable_raises(monkeypatch, config_file):
+def test_blocks_with_everything_set_carry_no_unresolved_record(monkeypatch, config_file):
     monkeypatch.setenv('MYSQL_HOST', 'db.example.com')
-    monkeypatch.setenv('DB_USER', '')
+    monkeypatch.setenv('DB_USER', 'iam_user')
     path = config_file(CONFIG)
 
-    with pytest.raises(ConfigError, match='DB_USER'):
-        utils.load_and_parse_config(config_file_path=path, configurations='databases')
+    config = utils.load_and_parse_config(config_file_path=path, configurations='databases')
+
+    assert '_unresolved_env' not in config['databases'][0]
+
+
+def test_an_unrelated_block_does_not_break_parsing(monkeypatch, config_file):
+    """The reported regression: one database's unset variable blocked every other database."""
+    monkeypatch.delenv('SNOW_PK_PASSPHRASE', raising=False)
+    path = config_file({'databases': [
+        {'name': 'snow_s', 'type': 'snowflake',
+         'private_key_passphrase': '${SNOW_PK_PASSPHRASE}'},
+        {'name': 'reporting_db', 'type': 'mysql', 'host': 'db.example.com',
+         'user': 'iam_user', 'database': 'd', 'iam_auth': True},
+    ]})
+
+    config = utils.load_and_parse_config(config_file_path=path, configurations='databases')
+
+    usable = config['databases'][1]
+    assert usable['user'] == 'iam_user'
+    assert '_unresolved_env' not in usable
+
+
+def test_describe_unresolved_names_the_variable_key_and_database():
+    message = utils.describe_unresolved('reporting_db', 'user', 'DB_USER')
+
+    assert 'DB_USER' in message
+    assert 'user' in message
+    assert 'reporting_db' in message
+    assert utils.CONFIG_FILENAME in message
 
 
 def test_placeholders_are_substituted_when_set(monkeypatch, config_file):

fullscript_spellbook-3.0.0/spellbook_config.yaml

@@ -1,54 +0,0 @@
-databases:
-  # Snowflake Database Connection
-  - name: alias_of_your_connection
-    type: snowflake
-    account: your_account
-    user: snowflake_user_name # Optional for private key auth
-    password: snowflake_password # Optional, used if private key is not present
-    private_key: snowflake_pk # Base64-encoded private key
-    private_key_passphrase: snowflake_pk_phrase # Optional
-    role: snowflake_role
-    warehouse: snowflake_warehouse
-    database: snowflake_database
-    schema: snowflake_schema
-
-  - name: alias_of_your_connection_2
-    type: snowflake
-    account: ${SNOWFLAKE_ACCOUNT}
-    user: ${SNOWFLAKE_USER} # Example of using environment variables
-    password: ${SNOWFLAKE_PASSWORD} # Example of using environment variables
-    private_key: snowflake_pk  # Example of using environment variables
-    private_key_passphrase: snowflake_pk_phrase # Optional
-    role: snowflake_role
-    warehouse: snowflake_warehouse
-    database: snowflake_database
-    schema: snowflake_schema
-
-  # MySQL Database Connection
-  - name: alias_of_your_connection
-    type: mysql
-    host: your_host
-    port: 3306
-    user: mysql_user_name
-    password: mysql_password
-    database: mysql_database
-
-  # MySQL Database Connection using AWS RDS IAM authentication.
-  # Requires the `rx` CLI, which is Fullscript internal tooling.
-  - name: alias_of_your_connection_3
-    type: mysql
-    host: ${MYSQL_HOST} # e.g. your-instance.abc123.us-east-1.rds.amazonaws.com
-    port: 3306
-    user: ${DB_USER} # Your IAM database user
-    iam_auth: true # Use an AWS RDS IAM token instead of a password
-    rds_db_name: your_rds_instance_name # Optional, as named by `rx rds list`; defaults to `name`
-    database: ${MYSQL_DATABASE}
-
-google_accounts:
-  # Google Sheets Service Account 1
-  - name: google_sheets_service_account_1
-    service_account: "your_base64_encoded_service_account_credentials_here"  # Add your Base64-encoded service account JSON here
-  # Google Sheets Service Account 2
-  - name: google_sheets_service_account_2
-    service_account: "your_base64_encoded_service_account_credentials_here"
-