nuvu-scan 2.0.2__tar.gz → 2.1.6__tar.gz

nuvu_scan-2.1.6/.cursorrules

@@ -0,0 +1,103 @@
+# Nuvu Scan - AI Agent Instructions
+
+## Project Overview
+Nuvu Scan is an open-source CLI tool for cloud data governance. It scans AWS and GCP to discover, govern, and optimize cloud data assets.
+
+## Critical Rules for AI Agents
+
+### 1. ALWAYS Write Tests for New Features
+Every new feature, CLI option, or code change MUST include corresponding tests:
+
+- **CLI changes**: Add tests in `tests/test_cli.py`
+- **Formatter changes**: Add tests in `tests/test_formatters.py`
+- **Scanner/collector changes**: Add tests in `tests/test_scanners.py`
+- **API/push changes**: Add tests in `tests/test_push_payload.py`
+- **New collectors**: Create `tests/test_<collector_name>.py`
+
+Tests run automatically on every commit via pre-commit hooks. Commits will be blocked if tests fail.
+
+### 2. Code Quality Standards
+- Use `ruff` for linting and formatting (NOT black)
+- Run `uv run ruff format .` before committing
+- Run `uv run ruff check .` to check for issues
+- Pre-commit hooks will auto-run: ruff, ruff-format, bandit, pytest
+
+### 3. CLI Option Changes
+When adding/modifying CLI options:
+1. Add `@click.option()` decorator in `nuvu_scan/cli/commands/scan.py`
+2. Add corresponding function parameter
+3. Add test in `tests/test_cli.py` to verify option exists
+4. Update `README.md` with usage examples
+
+### 4. Push Payload Format (API Compatibility)
+When modifying push functionality:
+- Payload MUST match the schema expected by `/api/scans/import`
+- Required fields: `provider`, `account_id`, `scan_timestamp`, `assets`, `total_cost_estimate_usd`
+- Asset fields: `provider`, `asset_type`, `normalized_category`, `region`, `arn`, `name`
+- Add tests in `tests/test_push_payload.py` to verify format
+
+### 5. Normalized Categories
+Use only these categories from `NormalizedCategory` enum:
+- OBJECT_STORAGE, DATA_WAREHOUSE, STREAMING, COMPUTE, ML_TRAINING
+- DATA_CATALOG, DATA_INTEGRATION, DATA_PIPELINE, DATA_SHARING
+- QUERY_ENGINE, SEARCH, DATABASE, SECURITY, BILLING
+
+### 6. Adding New Collectors
+When adding a new AWS/GCP collector:
+1. Create collector in `nuvu_scan/core/providers/<provider>/collectors/<name>.py`
+2. Register in the scanner's collector list
+3. Add to `--list-collectors` output
+4. Update `README.md` with new service coverage
+5. Create tests with mocked API responses
+6. Update IAM policy if new permissions needed
+
+### 7. Dependencies
+- Use `uv` for package management (NOT pip directly)
+- Add dependencies to `pyproject.toml`
+- Run `uv sync --dev` after adding dependencies
+
+### 8. Testing Commands
+```bash
+# Run all tests
+uv run pytest
+
+# Run with coverage
+uv run pytest --cov=nuvu_scan
+
+# Run specific test file
+uv run pytest tests/test_cli.py
+
+# Run pre-commit checks (includes tests)
+uv run pre-commit run --all-files
+```
+
+### 9. File Structure
+```
+nuvu_scan/
+├── cli/
+│   ├── commands/scan.py     # CLI commands and options
+│   └── formatters/          # HTML, JSON, CSV output
+├── core/
+│   ├── base.py              # Asset, ScanResult, NormalizedCategory
+│   └── providers/
+│       ├── aws/collectors/  # S3, Glue, Redshift, etc.
+│       └── gcp/collectors/  # GCS, BigQuery, etc.
+tests/
+├── test_cli.py              # CLI option tests
+├── test_formatters.py       # Output format tests
+├── test_scanners.py         # Scanner tests
+└── test_push_payload.py     # API payload tests
+```
+
+### 10. Commit Guidelines
+- Pre-commit hooks will run automatically
+- All tests must pass before commit is accepted
+- Use conventional commit messages: `feat:`, `fix:`, `test:`, `docs:`, `chore:`
+
+## Summary
+**Before any code change is complete, ensure:**
+1. ✅ Tests are written/updated
+2. ✅ `uv run pytest` passes
+3. ✅ `uv run ruff check .` passes
+4. ✅ README.md is updated (if user-facing)
+5. ✅ Pre-commit hooks pass on commit

{nuvu_scan-2.0.2 → nuvu_scan-2.1.6}/.gitignore

@@ -61,6 +61,9 @@ test-scan*.html
 *.json
 !tests/**/*.json
 
+# Raw scan data (contains sensitive AWS data)
+raws/
+
 # Environment variables
 .env
 .env.local

{nuvu_scan-2.0.2 → nuvu_scan-2.1.6}/.pre-commit-config.yaml

@@ -34,6 +34,17 @@ repos:
         args: ["-c", "pyproject.toml"]
         additional_dependencies: ["bandit[toml]"]
 
+  # Run tests before commit (only when Python files change)
+  - repo: local
+    hooks:
+      - id: pytest
+        name: pytest
+        entry: uv run pytest tests/ -x -q --tb=no
+        language: system
+        pass_filenames: false
+        types: [python]
+        stages: [pre-commit]
+
 # Configuration
 ci:
   autofix_commit_msg: |

{nuvu_scan-2.0.2 → nuvu_scan-2.1.6}/CONTRIBUTING.md

@@ -30,14 +30,22 @@ cd nuvu-scan
 uv sync --dev  # Creates .venv automatically, no activation needed!
 ```
 
-### 2. Make Changes
+### 2. Install Pre-commit Hooks
+
+```bash
+uv run pre-commit install
+```
+
+This ensures tests and linting run automatically on every commit.
+
+### 3. Make Changes
 
 - Write clear, readable code
-- Follow existing code style (enforced by black and ruff)
-- Add tests for new functionality
+- Follow existing code style (enforced by ruff)
+- **⚠️ Add tests for new functionality** (required - commits will fail without tests)
 - Update documentation
 
-### 3. Test Your Changes
+### 4. Test Your Changes
 
 ```bash
 # Run all tests (uv automatically uses .venv)
@@ -47,23 +55,30 @@ uv run pytest
 uv run pytest --cov=nuvu_scan
 
 # Check code quality
-uv run black .
+uv run ruff format .
 uv run ruff check .
 uv run mypy nuvu_scan
+
+# Run all pre-commit checks (recommended)
+uv run pre-commit run --all-files
 ```
 
 **Note**: No need to activate `.venv` - `uv run` handles it automatically!
 
-### 4. Commit
+### 5. Commit
+
+Pre-commit hooks will automatically run ruff, bandit, and pytest. If any check fails, the commit will be blocked.
 
-Use clear, descriptive commit messages:
+Use conventional commit messages:
 
 ```bash
-git commit -m "Add GCP BigQuery collector"
-git commit -m "Fix S3 bucket size calculation"
+git commit -m "feat: add GCP BigQuery collector"
+git commit -m "fix: correct S3 bucket size calculation"
+git commit -m "test: add tests for Redshift collector"
+git commit -m "docs: update CLI options in README"
 ```
 
-### 5. Push and Create PR
+### 6. Push and Create PR
 
 ```bash
 git push origin feature/your-feature
@@ -93,11 +108,23 @@ See the detailed guide in README.md under "Adding a New Cloud Provider".
 
 ## Code Style
 
-- **Formatting**: Use `black` (line length: 100)
-- **Linting**: Use `ruff`
+- **Formatting**: Use `ruff format`
+- **Linting**: Use `ruff check`
 - **Type hints**: Add type hints where helpful
 - **Docstrings**: Add docstrings for public functions/classes
 
+## Testing Requirements
+
+**Every new feature MUST include tests.** Pre-commit hooks run `pytest` automatically.
+
+| Change Type | Test File |
+|-------------|-----------|
+| CLI options | `tests/test_cli.py` |
+| Formatters (HTML/JSON/CSV) | `tests/test_formatters.py` |
+| Scanners/Collectors | `tests/test_scanners.py` |
+| Push/API changes | `tests/test_push_payload.py` |
+| New collector | `tests/test_<collector>.py` |
+
 ## Pull Request Process
 
 1. Ensure all tests pass

nuvu_scan-2.1.6/DEVELOPMENT_STATUS.md

@@ -0,0 +1,216 @@
+# Nuvu Scan - Development Status
+
+**Multi-Cloud Data Asset Scanner** - Open-source scanner designed to discover and inventory cloud data assets across AWS, GCP, Azure, and Databricks.
+
+> **Note**: nuvu-scan is a read-only scanner that collects asset metadata. For governance, policy enforcement, and decision-making, see [nuvu-cloud](https://github.com/nuvudev/nuvu-cloud).
+
+## ✅ Available Collectors (v2.1.0)
+
+### AWS Collectors (18 collectors)
+
+| Collector | Command Flag | What It Scans | Key Metrics |
+|-----------|--------------|---------------|-------------|
+| **S3** | `s3` | Buckets, policies, encryption | Size, public access, versioning |
+| **Glue** | `glue` | Databases, tables, crawlers, jobs, connections | Table counts, crawl status, job runs |
+| **Athena** | `athena` | Workgroups, query history | Query stats, failure rates |
+| **Redshift** | `redshift` | Clusters, serverless, snapshots, datashares, reserved nodes | CPU, connections, WLM, costs |
+| **IAM** | `iam` | Roles, users, groups, access keys | Permissions, MFA, key age, last used |
+| **MWAA** | `mwaa` | Apache Airflow environments | Environment class, worker counts |
+| **EC2/VPC** | `ec2` | Security groups, VPCs, instances, EBS volumes, Elastic IPs | Open ports, public IPs, volume encryption |
+| **KMS** | `kms` | Customer-managed encryption keys | Rotation status, key state |
+| **RDS** | `rds` | RDS instances, Aurora clusters, snapshots | Encryption, multi-AZ, backup retention |
+| **DynamoDB** | `dynamodb` | DynamoDB tables | PITR, encryption, capacity mode |
+| **Lambda** | `lambda` | Lambda functions | Runtime, code size, VPC config |
+| **Secrets Manager** | `secrets` | Secrets | Rotation, last accessed, age |
+| **AWS Backup** | `backup` | Backup vaults, backup plans | Recovery points, lifecycle |
+| **EKS** | `eks` | EKS clusters, node groups | K8s version, endpoint access |
+| **SNS/SQS** | `sns_sqs` | SNS topics, SQS queues | Encryption, DLQ, message counts |
+| **Lake Formation** | `lakeformation` | Data lake settings, permissions, LF-Tags | Permission grants, admin count |
+| **CloudTrail** | `cloudtrail` | CloudTrail trails | Multi-region, encryption, logging status |
+| **CloudWatch** | `cloudwatch` | CloudWatch log groups | Retention, encryption, size |
+
+### GCP Collectors (6 collectors)
+
+| Collector | Command Flag | What It Scans | Key Metrics |
+|-----------|--------------|---------------|-------------|
+| **GCS** | `gcs` | Cloud Storage buckets | Size, public access, lifecycle |
+| **BigQuery** | `bigquery` | Datasets, tables, query history | Table sizes, query costs |
+| **Dataproc** | `dataproc` | Dataproc clusters | Cluster config, job history |
+| **Pub/Sub** | `pubsub` | Topics, subscriptions | Message counts |
+| **IAM** | `iam` | Service accounts | Roles, permissions |
+| **Gemini** | `gemini` | Gemini API usage | API costs |
+
+## 📋 Usage
+
+### Basic Scan
+```bash
+# Scan all AWS collectors
+nuvu-scan aws
+
+# Scan all GCP collectors
+nuvu-scan gcp --credentials /path/to/key.json
+```
+
+### Selective Scanning
+```bash
+# Scan specific collectors
+nuvu-scan aws --collectors s3,rds,iam,kms
+
+# List available collectors
+nuvu-scan aws --list-collectors
+```
+
+### Output Formats
+```bash
+# HTML report (default)
+nuvu-scan aws -o report.html
+
+# JSON output
+nuvu-scan aws -o assets.json
+
+# CSV output
+nuvu-scan aws -o assets.csv
+```
+
+### Push to Nuvu Cloud
+```bash
+# Push results to nuvu-cloud for governance
+nuvu-scan aws --push --api-key YOUR_API_KEY
+```
+
+## 🔒 IAM Permissions
+
+The complete IAM policy is in `aws-iam-policy.json` (60+ permission statements).
+
+### Permission Categories
+
+| Category | Services | Example Actions |
+|----------|----------|-----------------|
+| **Storage** | S3, EBS | `s3:GetBucket*`, `ec2:DescribeVolumes` |
+| **Compute** | EC2, Lambda, EKS | `ec2:DescribeInstances`, `lambda:ListFunctions` |
+| **Database** | RDS, DynamoDB, Redshift | `rds:DescribeDB*`, `dynamodb:DescribeTable` |
+| **Data Analytics** | Glue, Athena, Lake Formation | `glue:GetTables`, `athena:ListWorkGroups` |
+| **Security** | IAM, KMS, Secrets Manager | `iam:ListRoles`, `kms:DescribeKey` |
+| **Networking** | VPC, Security Groups | `ec2:DescribeSecurityGroups`, `ec2:DescribeVpcs` |
+| **Messaging** | SNS, SQS | `sns:GetTopicAttributes`, `sqs:GetQueueAttributes` |
+| **Observability** | CloudWatch, CloudTrail | `logs:DescribeLogGroups`, `cloudtrail:DescribeTrails` |
+| **Resilience** | AWS Backup | `backup:ListBackupVaults`, `backup:ListBackupPlans` |
+| **Cost** | Cost Explorer | `ce:GetCostAndUsage` |
+
+All permissions are **read-only** following the principle of least privilege.
+
+## 📊 Asset Types Collected
+
+### Compute
+- EC2 instances, EBS volumes, Elastic IPs
+- Lambda functions
+- EKS clusters, node groups
+
+### Storage
+- S3 buckets
+- EBS volumes
+
+### Databases
+- RDS instances, Aurora clusters
+- DynamoDB tables
+- Redshift clusters (provisioned & serverless)
+
+### Data Catalog
+- Glue databases, tables, crawlers, jobs
+- Lake Formation settings, permissions, LF-Tags
+
+### Security
+- IAM roles, users, groups, access keys
+- KMS keys
+- Secrets Manager secrets
+- Security groups, VPCs
+
+### Observability
+- CloudWatch log groups
+- CloudTrail trails
+
+### Messaging
+- SNS topics
+- SQS queues
+
+### Backup
+- AWS Backup vaults and plans
+
+## 🏷️ Risk Flags
+
+nuvu-scan identifies potential issues by flagging assets:
+
+| Category | Example Flags |
+|----------|---------------|
+| **Security** | `unencrypted`, `publicly_accessible`, `mfa_disabled`, `open_to_internet` |
+| **Access** | `unused_role`, `old_key`, `overly_permissive`, `public_access` |
+| **Operations** | `no_backups`, `stale_crawler`, `deprecated_runtime`, `logging_disabled` |
+| **Cost** | `unattached_volume`, `old_snapshot`, `unused_eip` |
+| **Compliance** | `no_retention_policy`, `rotation_disabled`, `pitr_disabled` |
+
+## 🧪 Testing
+
+```bash
+# Run tests
+uv run pytest
+
+# Run with coverage
+uv run pytest --cov=nuvu_scan
+```
+
+## 📦 Installation
+
+```bash
+# From PyPI
+pip install nuvu-scan
+
+# From source
+git clone https://github.com/nuvudev/nuvu-scan
+cd nuvu-scan
+uv sync
+```
+
+## 📋 Roadmap
+
+### Additional AWS Collectors
+- [ ] OpenSearch collector
+- [ ] EMR collector
+- [ ] SageMaker collector
+- [ ] Bedrock collector
+- [ ] MSK (Kafka) collector
+- [ ] Kinesis collector
+- [ ] Step Functions collector
+- [ ] EventBridge collector
+
+### Additional GCP Collectors
+- [ ] Cloud SQL collector
+- [ ] Cloud Spanner collector
+- [ ] Bigtable collector
+- [ ] Firestore collector
+- [ ] Vertex AI collector
+- [ ] Dataflow collector
+- [ ] Cloud Composer collector
+
+### Azure Provider
+- [ ] Blob Storage collector
+- [ ] Data Lake collector
+- [ ] Synapse collector
+- [ ] Azure Databricks collector
+
+### Databricks Provider
+- [ ] Workspace discovery
+- [ ] Unity Catalog
+
+### Enhancements
+- [ ] Parallel collection for faster scans
+- [ ] Progress bars with ETA
+- [ ] Incremental scanning (delta detection)
+- [ ] Schema-level inventory (Redshift Data API)
+
+## 🤝 Contributing
+
+See [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.
+
+## 📄 License
+
+MIT License - see [LICENSE](LICENSE) for details.

{nuvu_scan-2.0.2 → nuvu_scan-2.1.6}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: nuvu-scan
-Version: 2.0.2
+Version: 2.1.6
 Summary: Multi-Cloud Data Asset Control - Discover, govern, and optimize your cloud data assets across AWS and GCP
 Project-URL: Homepage, https://nuvu.dev
 Project-URL: Documentation, https://github.com/nuvudev/nuvu-scan#readme
@@ -48,6 +48,10 @@ Description-Content-Type: text/markdown
 
 # Nuvu Scan
 
+[![PyPI Downloads](https://img.shields.io/pypi/dm/nuvu-scan?style=flat-square&label=downloads&color=667eea)](https://pypistats.org/packages/nuvu-scan)
+[![GitHub Stars](https://img.shields.io/github/stars/nuvudev/nuvu-scan?style=flat-square&label=stars&color=667eea)](https://github.com/nuvudev/nuvu-scan)
+[![PyPI Version](https://img.shields.io/pypi/v/nuvu-scan?style=flat-square&label=version&color=764ba2)](https://pypi.org/project/nuvu-scan/)
+
 **Take Control of Your Cloud Data Estate**
 Discover, govern, and optimize your cloud data assets across **AWS and GCP** — reduce wasted spend, enforce compliance, and gain full visibility into unused, idle, or risky resources.
 
@@ -79,28 +83,6 @@ pip install nuvu-scan
 
 ## Usage
 
-### Optional: Push results to Nuvu Cloud
-
-Nuvu Scan is fully open-source and runs standalone — no account required.
-If you want dashboards, team workflows, and long‑term history, you can optionally push results to Nuvu Cloud.
-
-```bash
-# Push results to Nuvu Cloud (optional)
-nuvu scan --provider aws --push --api-key your_nuvu_api_key
-
-# Or use environment variable
-export NUVU_API_KEY=your_nuvu_api_key
-nuvu scan --provider aws --push
-
-# Custom cloud URL (defaults to https://nuvu.dev)
-nuvu scan --provider aws --push --nuvu-cloud-url https://nuvu.dev
-```
-
-What this means for open‑source users:
-- You can keep everything local and export JSON/CSV/HTML.
-- No cloud credentials are ever sent to Nuvu Cloud — only scan results.
-- The data collected is identical whether you run locally or push.
-
 ### AWS Scanning
 
 **Prerequisites:** Create an IAM user or role with the read-only policy from `aws-iam-policy.json`. See the [AWS Setup](#aws-v1---available-now) section below for detailed instructions.
@@ -120,11 +102,14 @@ nuvu scan --provider aws \
   --access-key-id your-key \
   --secret-access-key your-secret
 
-# Output to JSON
-nuvu scan --provider aws --output-format json --output-file report.json
+# Output to HTML/JSON/CSV
+nuvu scan --provider aws --output-format html --output-file report.json
 
 # Scan specific regions
 nuvu scan --provider aws --region us-east-1 --region eu-west-1
+
+# Scan specific collector
+nuvu scan --provider aws --output-format html --collectors redshift --region us-west-1
 ```
 
 #### 2. Access Key + Secret Key + Session Token (Temporary Credentials)
@@ -203,6 +188,10 @@ You can optionally push scan results to a remote API for centralized tracking:
 ```bash
 # Push results to a remote endpoint
 nuvu scan --provider aws --push --api-key your-api-key --api-url https://your-api.example.com
+
+# Push results to NUVU Cloud for Data Goverance layer
+nuvu scan --provider aws --push --api-key your-api-key
+
 ```
 
 This is useful for integrating with your own data governance platforms or CI/CD pipelines.
@@ -364,6 +353,29 @@ Nuvu requires read-only access to your GCP project via a Service Account. The to
 ### Azure, Databricks (Coming Soon)
 Multi-cloud support is built into the architecture. Additional providers will be added in future releases.
 
+### Optional: Push results to Nuvu Cloud
+
+Nuvu Scan is fully open-source and runs standalone — no account required.
+If you want dashboards, team workflows, data estate time travel and long‑term history, you can optionally push results to Nuvu Cloud.
+
+```bash
+# Push results to Nuvu Cloud (optional)
+nuvu scan --provider aws --push --api-key your_nuvu_api_key
+
+# Or use environment variable
+export NUVU_API_KEY=your_nuvu_api_key
+nuvu scan --provider aws --push
+
+# Custom API URL (defaults to https://nuvu.dev)
+nuvu scan --provider aws --push --api-url https://your-api.example.com
+```
+
+What this means for open‑source users:
+- You can keep everything local and export JSON/CSV/HTML.
+- No cloud credentials are ever sent to Nuvu Cloud — only scan results.
+- The data collected is identical whether you run locally or push.
+
+
 ## License
 
 Apache 2.0
@@ -413,14 +425,17 @@ uv run pytest tests/test_s3_collector.py
 ### Code Quality
 
 ```bash
-# Format code with black
-uv run black .
+# Format code with ruff
+uv run ruff format .
 
 # Lint with ruff
 uv run ruff check .
 
 # Type checking with mypy
 uv run mypy nuvu_scan
+
+# Run all pre-commit checks (including tests)
+uv run pre-commit run --all-files
 ```
 
 ### Building the Package
@@ -503,11 +518,11 @@ git checkout -b fix/your-bug-description
 
 ### 3. Make Changes
 
-- Follow the existing code style (enforced by black and ruff)
-- Add tests for new features
+- Follow the existing code style (enforced by ruff)
+- **Add tests for new features** (required - pre-commit runs tests)
 - Update documentation as needed
 - Ensure all tests pass: `uv run pytest`
-- Run code quality checks: `uv run black . && uv run ruff check .`
+- Run code quality checks: `uv run ruff format . && uv run ruff check .`
 
 ### 4. Commit and Push