dbt-meta 0.1.0__tar.gz

dbt_meta-0.1.0/.gitignore

@@ -0,0 +1,43 @@
+# macOS
+.DS_Store
+.AppleDouble
+.LSOverride
+
+# Vim
+*.swp
+*.swo
+*~
+
+# Emacs
+*~
+\#*\#
+.\#*
+
+# Backup files
+*.bak
+*.backup
+
+# Logs
+*.log
+
+# Temporary files
+tmp/
+temp/
+
+# Ignore all dotfiles and dotfolders
+.*
+
+# But keep these
+!.gitignore
+
+# Python cache and test artifacts
+__pycache__/
+*.py[cod]
+*$py.class
+htmlcov/
+.coverage
+.pytest_cache/
+.benchmarks/
+
+# Project-specific development files
+CLAUDE.md*.json

dbt_meta-0.1.0/CHANGELOG.md

@@ -0,0 +1,109 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [0.1.0] - 2025-11-25
+
+### Added
+
+#### TOML Configuration Support
+- **Modern configuration files** with XDG Base Directory compliance
+  - Config locations: `./.dbt-meta.toml`, `~/.config/dbt-meta/config.toml`, `~/.dbt-meta.toml`
+  - Priority: CLI flags > TOML config > Environment variables > Defaults
+  - Template-based initialization with inline documentation
+  - TOML parsing: `tomllib` (Python 3.11+) or `tomli` (Python <3.11)
+
+#### Settings Management Commands
+- **`meta settings init`** - Create config file from template
+- **`meta settings show`** - Display current merged configuration (text or JSON)
+- **`meta settings validate`** - Validate config file syntax and values
+- **`meta settings path`** - Show path to active config file
+
+#### Configuration System
+- **`Config.from_toml()`** - Load configuration from TOML file
+- **`Config.from_config_or_env()`** - Load from TOML with env var fallback
+- **`Config.from_env()`** - Load from environment variables only
+- **`Config.find_config_file()`** - Auto-discover config file
+- Full configuration dataclass with type hints and validation
+- Automatic path expansion (~/ to home directory)
+- Boolean parsing with sensible defaults
+
+#### CLI Improvements
+- **`-h` short flag support** - Both `-h` and `--help` work for all commands
+  - Enabled via `context_settings={"help_option_names": ["-h", "--help"]}`
+  - Works for main app and all subcommands (settings, etc.)
+- **Simplified `schema` command output**
+  - Text mode: Just the full table name (e.g., `admirals-bi-dwh.core_client.client_info`)
+  - JSON mode: `{"model_name": "...", "full_name": "..."}`
+  - Optimized for shell scripting and AI consumption
+
+#### Username Sanitization
+- **Improved BigQuery dataset compatibility**
+  - Replaces ALL non-alphanumeric characters (not just dots/hyphens)
+  - Uses regex: `re.sub(r'[^a-zA-Z0-9_]', '_', username)`
+  - Examples: `pavel.filianin` → `pavel_filianin`, `user@example.com` → `user_example_com`
+  - Ensures valid BigQuery dataset names (letters, numbers, underscores only)
+
+#### Core Metadata Commands
+- **`info`** - Model summary (name, schema, table, materialization, tags)
+- **`schema`** - Full table name (database.schema.table)
+- **`path`** - Relative file path to .sql file
+- **`columns`** - Column names and types with catalog/BigQuery fallback
+- **`sql`** - Compiled SQL (default) or raw SQL with `--jinja` flag
+- **`docs`** - Column names, types, and descriptions
+- **`config`** - Full dbt config (partition_by, cluster_by, etc.)
+
+#### Dependency Navigation
+- **`deps`** - Dependencies by type (refs, sources, macros)
+- **`parents`** - Upstream dependencies (direct or all ancestors with `-a`)
+- **`children`** - Downstream dependencies (direct or all descendants with `-a`)
+
+#### Search and Discovery
+- **`list [pattern]`** - List all models (optionally filter by pattern)
+- **`search <query>`** - Search models by name or description
+
+#### Fallback System
+- **3-level fallback** - Production manifest → Dev manifest → Catalog.json
+- **Catalog.json support** - Fallback to catalog when manifest columns empty
+- **Environment variables** - `DBT_FALLBACK_TARGET`, `DBT_FALLBACK_CATALOG` (default: `true`)
+- **Intelligent warnings** - Automatic git change detection with helpful suggestions
+
+#### Output Modes
+- **`--json, -j`** - JSON output for all commands (AI-friendly)
+- **Rich formatted output** - Colored tables and text (default)
+- **Combined flags** - Use `-dj`, `-ajd`, `-jm` for faster typing
+
+#### Flexible Naming Configuration
+- **Production table naming** - `DBT_PROD_TABLE_NAME` (alias_or_name, name, alias)
+- **Production schema/database** - `DBT_PROD_SCHEMA_SOURCE` (config_or_model, model, config)
+- **Dev schema naming** - `DBT_DEV_SCHEMA` (direct override, default: `personal_{username}`)
+- **Username override** - `DBT_USER` (default: `$USER`)
+
+#### Manifest Discovery
+- **Auto-discovery** - Searches for manifest.json automatically
+- **Simple Mode** - Works out-of-box with `./target/manifest.json` after `dbt compile`
+- **Production Mode** - `DBT_PROD_MANIFEST_PATH` for central manifest location
+- **Dev Mode** - `DBT_DEV_MANIFEST_PATH` (default: `./target/manifest.json`)
+- **`-m, --manifest` flag** - Explicit manifest path override
+
+#### Testing
+- **416 comprehensive tests** - All passing
+- **91.67% code coverage** - Exceeds 90% requirement
+- Test categories: unit, integration, performance benchmarks
+- Edge case testing: empty strings, null values, special characters
+
+#### Performance
+- **LRU caching** - Manifest parser cached for sub-10ms responses
+- **orjson** - Fast JSON parsing (6-20x faster than stdlib)
+- **Lazy loading** - Manifest loaded only when needed
+
+#### Documentation
+- Comprehensive README with examples and use cases
+- Environment variables reference
+- CLAUDE.md for AI agent integration
+- Apache 2.0 license
+
+[0.1.0]: https://github.com/Filianin/dbt-meta/releases/tag/v0.1.0

dbt_meta-0.1.0/CLAUDE.md

@@ -0,0 +1,492 @@
+# CLAUDE.md
+
+## Project Overview
+
+**dbt-meta** is an AI-first CLI tool for extracting metadata from dbt's `manifest.json`.
+
+**Key Design Principles:**
+- Performance-first: LRU caching, orjson parser, lazy loading
+- AI-optimized: JSON output mode, deterministic responses
+- Production-first: Automatically prioritizes production manifest
+- Fallback-enabled: BigQuery fallback when models missing from manifest
+
+## Development Setup
+
+```bash
+# Install in development mode
+pip install -e ".[dev]"
+
+# Run tests (95%+ coverage required)
+pytest --cov=dbt_meta
+
+# Type checking + linting
+mypy src/dbt_meta && ruff check src/dbt_meta
+```
+
+## Development Guidelines
+
+### File Management
+
+- **NO temporary files in `/tmp/`** - Save all files in project root instead
+- Temporary files should be visible in git (easy to review and discard)
+- Test scripts, debug files, analysis files - all go in project root
+- Example: `./test_catalog_fallback.sh` instead of `/tmp/test_catalog_fallback.sh`
+
+## Architecture
+
+### Module Structure
+
+```
+src/dbt_meta/
+├── cli.py                # Typer CLI + Rich formatting
+├── commands.py           # Command implementations + BigQuery fallback
+├── errors.py             # Exception hierarchy
+├── config.py             # Configuration management
+├── fallback.py           # 3-level fallback strategy
+├── utils/                # Utility modules
+│   ├── __init__.py       # Parser caching, warnings
+│   └── git.py            # Git operations
+└── manifest/
+    ├── parser.py         # Fast manifest parsing (orjson + caching)
+    └── finder.py         # 4-level manifest discovery
+```
+
+### Key Patterns
+
+#### 1. Three-Level Caching Strategy
+
+```python
+# Level 1: Parser instance caching (commands.py:20-34)
+@lru_cache(maxsize=1)
+def _get_cached_parser(manifest_path: str) -> ManifestParser
+
+# Level 2: Manifest lazy loading (manifest/parser.py:28-58)
+@cached_property
+def manifest(self) -> Dict[str, Any]
+
+# Level 3: orjson for fast parsing (6-20x faster than stdlib)
+```
+
+**Result:** Sub-10ms response times after first command.
+
+**CRITICAL:** Always use `_get_cached_parser()`, never instantiate `ManifestParser` directly.
+
+#### 2. Manifest Discovery (3-level priority)
+
+```
+1. --manifest PATH (explicit CLI flag - highest priority)
+2. DBT_DEV_MANIFEST_PATH (when --dev flag used, default: ./target/manifest.json)
+3. DBT_PROD_MANIFEST_PATH (production, default: ~/dbt-state/manifest.json)
+```
+
+**Critical distinction:**
+- Production manifest uses `config.alias` for table names
+- Dev manifest uses SQL filename for table names
+- When both `--manifest` and `--dev` are used, `--dev` is ignored with a warning
+- Always use `--dev` flag for dev tables
+
+#### 3. BigQuery Fallback Pattern
+
+```python
+model = parser.get_model(model_name)
+if not model:
+    if os.environ.get('DBT_FALLBACK_BIGQUERY', 'true').lower() in ('true', '1', 'yes'):
+        dataset, table = _infer_table_parts(model_name)
+        bq_metadata = _fetch_table_metadata_from_bigquery(dataset, table)
+        # Return partial metadata with warning to stderr
+```
+
+**Supported:** `schema`, `columns`, `info`, `config`
+**Not supported:** `deps`, `sql`, `parents`, `children` (dbt-specific)
+
+#### 4. Dev Schema Resolution (2-level priority)
+
+**Dev schema resolution (2-level):**
+
+```python
+1. DBT_DEV_SCHEMA - Direct schema name (highest priority)
+2. Default: "personal_{username}"
+```
+
+**Backward compatibility:** Old variables (`DBT_DEV_DATASET`, `DBT_DEV_SCHEMA_TEMPLATE`, `DBT_DEV_SCHEMA_PREFIX`) show deprecation warning.
+
+Location: `config.py:28-35`, `utils/dev.py:81-93`
+
+#### 5. Exception Hierarchy
+
+**Consistent error handling with typed exceptions**:
+
+```python
+# src/dbt_meta/errors.py
+
+DbtMetaError (base)
+├── ModelNotFoundError        # Model not in manifest/BigQuery
+├── ManifestNotFoundError     # manifest.json not found
+├── ManifestParseError        # Invalid JSON in manifest
+├── BigQueryError             # BigQuery operation failed
+├── GitOperationError         # Git command failed
+└── ConfigurationError        # Invalid configuration
+```
+
+**All exceptions include:**
+- `message`: Human-readable error description
+- `suggestion`: Actionable fix (optional)
+- Structured data for programmatic handling
+
+**CLI error handling** (`cli.py:45-66`):
+```python
+try:
+    result = commands.schema(manifest_path, model_name)
+    # ... handle result
+except DbtMetaError as e:
+    handle_error(e)  # Rich formatted output with suggestion
+```
+
+**Example error output:**
+```
+Error: Model 'core__clients' not found
+
+Suggestion: Searched in: production manifest, dev manifest
+Try: meta list core
+```
+
+**Benefits:**
+- Consistent error messages across all commands
+- Actionable suggestions for users
+- Easy to catch and handle in tests
+- AI-friendly structured errors
+
+#### 6. Configuration Management
+
+**Centralized configuration with TOML and env var support** (v0.1.0):
+
+```python
+# src/dbt_meta/config.py
+
+from dbt_meta.config import Config
+
+# Load from TOML config file (recommended)
+config = Config.from_config_or_env()
+# Priority: TOML config > Environment variables > Defaults
+# Searches: ./.dbt-meta.toml, ~/.config/dbt-meta/config.toml, ~/.dbt-meta.toml
+
+# Load from environment variables only
+config = Config.from_env()
+
+# Load from TOML file directly
+config = Config.from_toml("/path/to/config.toml")
+
+# Access configuration
+config.prod_manifest_path       # ~/dbt-state/manifest.json
+config.dev_manifest_path        # ./target/manifest.json
+config.fallback_dev_enabled     # True/False
+config.fallback_bigquery_enabled # True/False
+config.dev_dataset              # personal_username (sanitized)
+config.prod_table_name_strategy # alias_or_name | name | alias
+config.prod_schema_source       # config_or_model | model | config
+
+# Validate configuration
+warnings = config.validate()
+for warning in warnings:
+    print(f"Warning: {warning}")
+```
+
+**Key features:**
+- **TOML configuration** - Modern config files with XDG compliance
+- **Priority system** - CLI flags > TOML > Env vars > Defaults
+- Single source of truth for all configuration
+- Automatic path expansion (~ to home directory)
+- Boolean parsing with sensible defaults
+- Validation with helpful warnings
+- Type-safe dataclass with full type hints
+- Username sanitization for BigQuery compatibility (replaces all non-alphanumeric chars)
+
+**Dev schema resolution** (2-level):
+```python
+# Priority 1: Direct schema name
+DBT_DEV_SCHEMA = "my_custom_dev_schema"
+
+# Priority 2: Default with username (fallback)
+# personal_{username} (from USER env var, sanitized with re.sub(r'[^a-zA-Z0-9_]', '_', username))
+```
+
+**Config file locations** (priority order):
+1. `./.dbt-meta.toml` - Project-local config
+2. `~/.config/dbt-meta/config.toml` - User config (XDG)
+3. `~/.dbt-meta.toml` - Fallback
+
+**Settings commands** (v0.1.0):
+- `meta settings init` - Create config file from template
+- `meta settings show` - Display merged configuration
+- `meta settings validate` - Validate config file
+- `meta settings path` - Show active config file path
+
+Location: `config.py:24-473`
+
+#### 7. CLI and User Experience
+
+**Help system improvements** (v0.1.0):
+```python
+# Enable -h short flag for all commands and subcommands
+app = typer.Typer(
+    name="dbt-meta",
+    context_settings={"help_option_names": ["-h", "--help"]},
+)
+
+settings_app = typer.Typer(
+    help="CLI settings management",
+    context_settings={"help_option_names": ["-h", "--help"]},
+)
+```
+
+**Benefits:**
+- Both `-h` and `--help` work for all commands
+- Consistent UX across main app and subcommands
+- Standard CLI convention
+
+**Schema command output** (v0.1.0):
+```bash
+# Text mode - simple table name
+meta schema core_client__client_info
+# → admirals-bi-dwh.core_client.client_info
+
+# JSON mode - structured output
+meta schema -j core_client__client_info
+# → {"model_name": "core_client__client_info", "full_name": "admirals-bi-dwh.core_client.client_info"}
+```
+
+**Purpose:** Minimalist output optimized for shell scripting and AI consumption
+
+**Username sanitization** (v0.1.0):
+```python
+# config.py:36-57 - _calculate_dev_schema()
+username_sanitized = re.sub(r'[^a-zA-Z0-9_]', '_', username)
+# Replaces ALL non-alphanumeric characters (dots, hyphens, @, etc.)
+# Examples:
+# "pavel.filianin" → "pavel_filianin"
+# "john-doe" → "john_doe"
+# "user@example.com" → "user_example_com"
+```
+
+**Why:** BigQuery dataset names only allow letters, numbers, and underscores
+
+Location: `cli.py:26-39`, `config.py:36-57`
+
+#### 8. Fallback Strategy
+
+**3-level fallback system with clean interface**:
+
+```python
+# src/dbt_meta/fallback.py
+
+from dbt_meta.fallback import FallbackStrategy, FallbackLevel, FallbackResult
+from dbt_meta.config import Config
+
+config = Config.from_env()
+strategy = FallbackStrategy(config)
+
+# Try to get model with automatic fallback
+result = strategy.get_model(
+    model_name="core__clients",
+    prod_parser=parser,
+    allowed_levels=[
+        FallbackLevel.PROD_MANIFEST,
+        FallbackLevel.DEV_MANIFEST,
+        FallbackLevel.BIGQUERY  # Optional - exclude for deps/sql commands
+    ]
+)
+
+if result.found:
+    print(f"Found in: {result.level.value}")
+    print(f"Data: {result.data}")
+
+    # Show warnings (e.g., "Using dev manifest")
+    for warning in result.warnings:
+        print(f"Warning: {warning}")
+else:
+    # ModelNotFoundError raised if not found
+    pass
+```
+
+**Fallback levels (in priority order):**
+1. `PROD_MANIFEST` - Production manifest (default source)
+2. `DEV_MANIFEST` - Dev manifest (if enabled via `DBT_FALLBACK_TARGET`)
+3. `BIGQUERY` - BigQuery metadata (if enabled via `DBT_FALLBACK_BIGQUERY`)
+
+**Key features:**
+- Consolidates logic previously duplicated across 10+ commands
+- Automatic warning collection at each level
+- Configurable allowed levels per command
+- Clean error handling with `ModelNotFoundError`
+- Type-safe enums and dataclasses
+
+**Usage pattern for commands:**
+```python
+# commands with BigQuery support (schema, columns, info, config)
+allowed_levels = [FallbackLevel.PROD_MANIFEST, FallbackLevel.DEV_MANIFEST, FallbackLevel.BIGQUERY]
+
+# commands without BigQuery support (deps, sql, parents, children)
+allowed_levels = [FallbackLevel.PROD_MANIFEST, FallbackLevel.DEV_MANIFEST]
+```
+
+Location: `fallback.py:18-198`
+
+**Note:** BigQuery fallback (`_fetch_from_bigquery`) is currently a placeholder (returns None). Full implementation will be added when refactoring `commands.py` in Task 3.
+
+## Adding a New Command
+
+### 1. Add command function in `commands.py`
+
+```python
+def new_command(manifest_path: str, model_name: str) -> Optional[Dict]:
+    """Extract metadata"""
+    parser = _get_cached_parser(manifest_path)  # MUST use cached parser
+    model = parser.get_model(model_name)
+
+    if not model:
+        return None  # Or add BigQuery fallback if applicable
+
+    return {'field': model.get('field', '')}
+```
+
+### 2. Add CLI command in `cli.py`
+
+```python
+@app.command()
+def new_command(
+    model_name: str = typer.Argument(..., help="Model name"),
+    json_output: bool = typer.Option(False, "-j", "--json"),
+):
+    """Command description"""
+    manifest_path = get_manifest_path()
+    result = commands.new_command(manifest_path, model_name)
+    handle_command_output(result, json_output)
+```
+
+### 3. Add tests in `test_commands.py`
+
+```python
+def test_new_command(prod_manifest, test_model):
+    result = commands.new_command(prod_manifest, test_model)
+    assert result is not None
+    assert 'field' in result
+```
+
+### 4. Update help text in `cli.py`
+
+Add to `_build_commands_panel()` if needed.
+
+## Testing Strategy
+
+**Coverage requirement:** 95%+ (pyproject.toml:79)
+
+**Test markers:**
+- `@pytest.mark.unit` - Fast unit tests
+- `@pytest.mark.integration` - Integration tests
+- `@pytest.mark.performance` - Performance benchmarks
+
+**Test structure:**
+- `test_commands.py` - Command implementations
+- `test_infrastructure.py` - Manifest discovery + warnings
+- `test_errors.py` - Exception hierarchy
+- `test_config.py` - Configuration management
+- `test_fallback.py` - Fallback strategy
+- `conftest.py` - Shared fixtures (uses dynamic `test_model` fixture)
+
+**Excluded from coverage:**
+- `cli.py` - UI layer (tested manually)
+- `manifest/finder.py` - Auto-discovery logic
+
+## Important Code Locations
+
+| Feature | Location |
+|---------|----------|
+| Exception hierarchy | `errors.py:13-203` |
+| Error handler (CLI) | `cli.py:45-66` |
+| Configuration management | `config.py:12-139` |
+| Fallback strategy | `fallback.py:18-198` |
+| Manifest discovery | `manifest/finder.py:26-89` |
+| Parser caching | `commands.py:20-34`, `manifest/parser.py:28-58` |
+| BigQuery fallback | `commands.py:399-446` |
+| Dev schema resolution | `commands.py:934-1042` (deprecated, use `config.py`) |
+| Prod table naming | `commands.py:452-493` |
+| Lineage traversal | `commands.py:773-805` |
+| Help formatting | `cli.py:43-157` |
+
+## Environment Variables
+
+**Preferred access:** Use `Config.from_env()` for centralized configuration management with validation.
+
+**Manifest:**
+- `DBT_PROD_MANIFEST_PATH` - Production manifest path (default: `~/.dbt-state/manifest.json`)
+- `DBT_DEV_MANIFEST_PATH` - Dev manifest path (default: `./target/manifest.json`)
+
+**Fallback control:**
+- `DBT_FALLBACK_TARGET` - Enable dev manifest fallback (default: `true`)
+- `DBT_FALLBACK_BIGQUERY` - Enable BigQuery fallback (default: `true`)
+
+**Naming:**
+- `DBT_PROD_TABLE_NAME` - `alias_or_name` (default), `name`, `alias`
+- `DBT_PROD_SCHEMA_SOURCE` - `config_or_model` (default), `model`, `config`
+- `DBT_DEV_SCHEMA` - Direct dev schema name (overrides default `personal_{username}`)
+- `DBT_USER` - Override username for dev schema (default: `$USER`)
+
+**Deprecated:**
+- `DBT_DEV_DATASET` - Use `DBT_DEV_SCHEMA` instead (backward compatible with warning)
+- `DBT_DEV_SCHEMA_TEMPLATE` - Use `DBT_DEV_SCHEMA` instead (no longer supported)
+- `DBT_DEV_SCHEMA_PREFIX` - Use `DBT_DEV_SCHEMA` instead (no longer supported)
+
+## Type Checking
+
+**Strict mode enabled** - All functions must have type hints.
+
+```python
+from typing import Dict, List, Optional, Any
+
+def command(manifest_path: str, model_name: str) -> Optional[Dict[str, Any]]:
+    """Returns None if model not found"""
+    ...
+
+def search(manifest_path: str, query: str) -> List[Dict[str, str]]:
+    """Always returns list (empty if no results)"""
+    ...
+```
+
+## Data Source Decision Logic
+
+For understanding how dbt-meta determines which data source to use (prod/dev manifest, BigQuery fallback):
+
+**Reference documentation:**
+- `.qa/decision_tree_visual.txt` - Visual decision tree with 5 critical scenarios
+- `.qa/data_source_logic.md` - Detailed logic specification
+
+**Key principle:**
+When BigQuery fallback is needed (empty columns), ALWAYS use schema from the FOUND model:
+- Model found in dev manifest → use dev schema (`personal_user`)
+- Model found in prod manifest → use prod schema
+- Never re-search production manifest after model is found
+
+**Implementation:**
+- `command_impl/base.py:107-151` - Main fallback orchestration
+- `command_impl/columns.py:71-87` - BigQuery fallback with correct schema
+- `utils/git.py:77-169` - Git status detection and warnings
+
+## Publishing Checklist
+
+1. `pytest && mypy src/dbt_meta && ruff check src/dbt_meta`
+2. Update version in `pyproject.toml`, `src/dbt_meta/__init__.py`
+3. Update `CHANGELOG.md` with version and date
+4. Test: `pip install -e . && meta --version`
+5. Tag: `git tag v0.x.0`
+
+## Performance Benchmarks
+
+- First command: 30-60ms (manifest parsing + caching)
+- Subsequent commands: 5-10ms (cached parser)
+- 865+ models parsed in ~35ms median
+
+**Optimization rules:**
+1. Always use `_get_cached_parser()` - never instantiate `ManifestParser` directly
+2. Cache results in local variables - avoid repeated manifest access
+3. Use generator expressions over list comprehensions when possible
+4. Use `@lru_cache` for expensive helper functions