nuvu-scan 2.1.2__tar.gz → 2.1.6__tar.gz

{nuvu_scan-2.1.2 → nuvu_scan-2.1.6}/.gitignore

@@ -61,6 +61,9 @@ test-scan*.html
 *.json
 !tests/**/*.json
 
+# Raw scan data (contains sensitive AWS data)
+raws/
+
 # Environment variables
 .env
 .env.local

nuvu_scan-2.1.6/DEVELOPMENT_STATUS.md

@@ -0,0 +1,216 @@
+# Nuvu Scan - Development Status
+
+**Multi-Cloud Data Asset Scanner** - Open-source scanner designed to discover and inventory cloud data assets across AWS, GCP, Azure, and Databricks.
+
+> **Note**: nuvu-scan is a read-only scanner that collects asset metadata. For governance, policy enforcement, and decision-making, see [nuvu-cloud](https://github.com/nuvudev/nuvu-cloud).
+
+## ✅ Available Collectors (v2.1.0)
+
+### AWS Collectors (18 collectors)
+
+| Collector | Command Flag | What It Scans | Key Metrics |
+|-----------|--------------|---------------|-------------|
+| **S3** | `s3` | Buckets, policies, encryption | Size, public access, versioning |
+| **Glue** | `glue` | Databases, tables, crawlers, jobs, connections | Table counts, crawl status, job runs |
+| **Athena** | `athena` | Workgroups, query history | Query stats, failure rates |
+| **Redshift** | `redshift` | Clusters, serverless, snapshots, datashares, reserved nodes | CPU, connections, WLM, costs |
+| **IAM** | `iam` | Roles, users, groups, access keys | Permissions, MFA, key age, last used |
+| **MWAA** | `mwaa` | Apache Airflow environments | Environment class, worker counts |
+| **EC2/VPC** | `ec2` | Security groups, VPCs, instances, EBS volumes, Elastic IPs | Open ports, public IPs, volume encryption |
+| **KMS** | `kms` | Customer-managed encryption keys | Rotation status, key state |
+| **RDS** | `rds` | RDS instances, Aurora clusters, snapshots | Encryption, multi-AZ, backup retention |
+| **DynamoDB** | `dynamodb` | DynamoDB tables | PITR, encryption, capacity mode |
+| **Lambda** | `lambda` | Lambda functions | Runtime, code size, VPC config |
+| **Secrets Manager** | `secrets` | Secrets | Rotation, last accessed, age |
+| **AWS Backup** | `backup` | Backup vaults, backup plans | Recovery points, lifecycle |
+| **EKS** | `eks` | EKS clusters, node groups | K8s version, endpoint access |
+| **SNS/SQS** | `sns_sqs` | SNS topics, SQS queues | Encryption, DLQ, message counts |
+| **Lake Formation** | `lakeformation` | Data lake settings, permissions, LF-Tags | Permission grants, admin count |
+| **CloudTrail** | `cloudtrail` | CloudTrail trails | Multi-region, encryption, logging status |
+| **CloudWatch** | `cloudwatch` | CloudWatch log groups | Retention, encryption, size |
+
+### GCP Collectors (6 collectors)
+
+| Collector | Command Flag | What It Scans | Key Metrics |
+|-----------|--------------|---------------|-------------|
+| **GCS** | `gcs` | Cloud Storage buckets | Size, public access, lifecycle |
+| **BigQuery** | `bigquery` | Datasets, tables, query history | Table sizes, query costs |
+| **Dataproc** | `dataproc` | Dataproc clusters | Cluster config, job history |
+| **Pub/Sub** | `pubsub` | Topics, subscriptions | Message counts |
+| **IAM** | `iam` | Service accounts | Roles, permissions |
+| **Gemini** | `gemini` | Gemini API usage | API costs |
+
+## 📋 Usage
+
+### Basic Scan
+```bash
+# Scan all AWS collectors
+nuvu-scan aws
+
+# Scan all GCP collectors
+nuvu-scan gcp --credentials /path/to/key.json
+```
+
+### Selective Scanning
+```bash
+# Scan specific collectors
+nuvu-scan aws --collectors s3,rds,iam,kms
+
+# List available collectors
+nuvu-scan aws --list-collectors
+```
+
+### Output Formats
+```bash
+# HTML report (default)
+nuvu-scan aws -o report.html
+
+# JSON output
+nuvu-scan aws -o assets.json
+
+# CSV output
+nuvu-scan aws -o assets.csv
+```
+
+### Push to Nuvu Cloud
+```bash
+# Push results to nuvu-cloud for governance
+nuvu-scan aws --push --api-key YOUR_API_KEY
+```
+
+## 🔒 IAM Permissions
+
+The complete IAM policy is in `aws-iam-policy.json` (60+ permission statements).
+
+### Permission Categories
+
+| Category | Services | Example Actions |
+|----------|----------|-----------------|
+| **Storage** | S3, EBS | `s3:GetBucket*`, `ec2:DescribeVolumes` |
+| **Compute** | EC2, Lambda, EKS | `ec2:DescribeInstances`, `lambda:ListFunctions` |
+| **Database** | RDS, DynamoDB, Redshift | `rds:DescribeDB*`, `dynamodb:DescribeTable` |
+| **Data Analytics** | Glue, Athena, Lake Formation | `glue:GetTables`, `athena:ListWorkGroups` |
+| **Security** | IAM, KMS, Secrets Manager | `iam:ListRoles`, `kms:DescribeKey` |
+| **Networking** | VPC, Security Groups | `ec2:DescribeSecurityGroups`, `ec2:DescribeVpcs` |
+| **Messaging** | SNS, SQS | `sns:GetTopicAttributes`, `sqs:GetQueueAttributes` |
+| **Observability** | CloudWatch, CloudTrail | `logs:DescribeLogGroups`, `cloudtrail:DescribeTrails` |
+| **Resilience** | AWS Backup | `backup:ListBackupVaults`, `backup:ListBackupPlans` |
+| **Cost** | Cost Explorer | `ce:GetCostAndUsage` |
+
+All permissions are **read-only** following the principle of least privilege.
+
+## 📊 Asset Types Collected
+
+### Compute
+- EC2 instances, EBS volumes, Elastic IPs
+- Lambda functions
+- EKS clusters, node groups
+
+### Storage
+- S3 buckets
+- EBS volumes
+
+### Databases
+- RDS instances, Aurora clusters
+- DynamoDB tables
+- Redshift clusters (provisioned & serverless)
+
+### Data Catalog
+- Glue databases, tables, crawlers, jobs
+- Lake Formation settings, permissions, LF-Tags
+
+### Security
+- IAM roles, users, groups, access keys
+- KMS keys
+- Secrets Manager secrets
+- Security groups, VPCs
+
+### Observability
+- CloudWatch log groups
+- CloudTrail trails
+
+### Messaging
+- SNS topics
+- SQS queues
+
+### Backup
+- AWS Backup vaults and plans
+
+## 🏷️ Risk Flags
+
+nuvu-scan identifies potential issues by flagging assets:
+
+| Category | Example Flags |
+|----------|---------------|
+| **Security** | `unencrypted`, `publicly_accessible`, `mfa_disabled`, `open_to_internet` |
+| **Access** | `unused_role`, `old_key`, `overly_permissive`, `public_access` |
+| **Operations** | `no_backups`, `stale_crawler`, `deprecated_runtime`, `logging_disabled` |
+| **Cost** | `unattached_volume`, `old_snapshot`, `unused_eip` |
+| **Compliance** | `no_retention_policy`, `rotation_disabled`, `pitr_disabled` |
+
+## 🧪 Testing
+
+```bash
+# Run tests
+uv run pytest
+
+# Run with coverage
+uv run pytest --cov=nuvu_scan
+```
+
+## 📦 Installation
+
+```bash
+# From PyPI
+pip install nuvu-scan
+
+# From source
+git clone https://github.com/nuvudev/nuvu-scan
+cd nuvu-scan
+uv sync
+```
+
+## 📋 Roadmap
+
+### Additional AWS Collectors
+- [ ] OpenSearch collector
+- [ ] EMR collector
+- [ ] SageMaker collector
+- [ ] Bedrock collector
+- [ ] MSK (Kafka) collector
+- [ ] Kinesis collector
+- [ ] Step Functions collector
+- [ ] EventBridge collector
+
+### Additional GCP Collectors
+- [ ] Cloud SQL collector
+- [ ] Cloud Spanner collector
+- [ ] Bigtable collector
+- [ ] Firestore collector
+- [ ] Vertex AI collector
+- [ ] Dataflow collector
+- [ ] Cloud Composer collector
+
+### Azure Provider
+- [ ] Blob Storage collector
+- [ ] Data Lake collector
+- [ ] Synapse collector
+- [ ] Azure Databricks collector
+
+### Databricks Provider
+- [ ] Workspace discovery
+- [ ] Unity Catalog
+
+### Enhancements
+- [ ] Parallel collection for faster scans
+- [ ] Progress bars with ETA
+- [ ] Incremental scanning (delta detection)
+- [ ] Schema-level inventory (Redshift Data API)
+
+## 🤝 Contributing
+
+See [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.
+
+## 📄 License
+
+MIT License - see [LICENSE](LICENSE) for details.

{nuvu_scan-2.1.2 → nuvu_scan-2.1.6}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: nuvu-scan
-Version: 2.1.2
+Version: 2.1.6
 Summary: Multi-Cloud Data Asset Control - Discover, govern, and optimize your cloud data assets across AWS and GCP
 Project-URL: Homepage, https://nuvu.dev
 Project-URL: Documentation, https://github.com/nuvudev/nuvu-scan#readme
@@ -48,6 +48,10 @@ Description-Content-Type: text/markdown
 
 # Nuvu Scan
 
+[![PyPI Downloads](https://img.shields.io/pypi/dm/nuvu-scan?style=flat-square&label=downloads&color=667eea)](https://pypistats.org/packages/nuvu-scan)
+[![GitHub Stars](https://img.shields.io/github/stars/nuvudev/nuvu-scan?style=flat-square&label=stars&color=667eea)](https://github.com/nuvudev/nuvu-scan)
+[![PyPI Version](https://img.shields.io/pypi/v/nuvu-scan?style=flat-square&label=version&color=764ba2)](https://pypi.org/project/nuvu-scan/)
+
 **Take Control of Your Cloud Data Estate**
 Discover, govern, and optimize your cloud data assets across **AWS and GCP** — reduce wasted spend, enforce compliance, and gain full visibility into unused, idle, or risky resources.
 

{nuvu_scan-2.1.2 → nuvu_scan-2.1.6}/README.md

@@ -1,5 +1,9 @@
 # Nuvu Scan
 
+[![PyPI Downloads](https://img.shields.io/pypi/dm/nuvu-scan?style=flat-square&label=downloads&color=667eea)](https://pypistats.org/packages/nuvu-scan)
+[![GitHub Stars](https://img.shields.io/github/stars/nuvudev/nuvu-scan?style=flat-square&label=stars&color=667eea)](https://github.com/nuvudev/nuvu-scan)
+[![PyPI Version](https://img.shields.io/pypi/v/nuvu-scan?style=flat-square&label=version&color=764ba2)](https://pypi.org/project/nuvu-scan/)
+
 **Take Control of Your Cloud Data Estate**
 Discover, govern, and optimize your cloud data assets across **AWS and GCP** — reduce wasted spend, enforce compliance, and gain full visibility into unused, idle, or risky resources.
 

{nuvu_scan-2.1.2 → nuvu_scan-2.1.6}/nuvu_scan/cli/commands/scan.py

@@ -331,6 +331,7 @@ def scan_command(
                 "total_cost_estimate_usd": result.total_cost_estimate_usd,
                 "scan_regions": scan_regions if scan_regions else None,
                 "scan_all_regions": not bool(region),
+                "summary": result.summary,  # Include cost data from Cost Explorer
                 "assets": [
                     {
                         "provider": asset.provider,
@@ -353,6 +354,7 @@ def scan_command(
                         "risk_flags": asset.risk_flags,
                         "ownership_confidence": asset.ownership_confidence or "unknown",
                         "suggested_owner": asset.suggested_owner,
+                        "underlying_cloud_account_id": asset.underlying_cloud_account_id,
                     }
                     for asset in result.assets
                 ],

{nuvu_scan-2.1.2 → nuvu_scan-2.1.6}/nuvu_scan/core/base.py

@@ -28,6 +28,16 @@ class NormalizedCategory(str, Enum):
     DATABASE = "database"
     SECURITY = "security"
     BILLING = "billing"
+    # Additional categories for comprehensive coverage
+    NETWORKING = "networking"  # VPC, Load Balancers, Route 53, CloudFront
+    CACHING = "caching"  # ElastiCache, DAX
+    CONTAINER = "container"  # ECS, ECR, Fargate
+    SERVERLESS = "serverless"  # Lambda, Step Functions, API Gateway
+    STORAGE = "storage"  # EBS, EFS, FSx
+    MESSAGING = "messaging"  # SNS, SQS, EventBridge
+    OBSERVABILITY = "observability"  # CloudWatch, X-Ray
+    RESILIENCE = "resilience"  # Backup, DR
+    GOVERNANCE = "governance"  # Config, CloudTrail, Organizations
 
 
 @dataclass

{nuvu_scan-2.1.2 → nuvu_scan-2.1.6}/nuvu_scan/core/providers/aws/aws_scanner.py

@@ -15,15 +15,37 @@ from nuvu_scan.core.base import (
     ScanConfig,
 )
 
+from .collectors.apigateway import APIGatewayCollector
 from .collectors.athena import AthenaCollector
+from .collectors.backup import BackupCollector
+from .collectors.cloudfront import CloudFrontCollector
+from .collectors.cloudtrail import CloudTrailCollector
+from .collectors.cloudwatch import CloudWatchLogsCollector
 from .collectors.cost_explorer import CostExplorerCollector
+from .collectors.dynamodb import DynamoDBCollector
+from .collectors.ec2 import EC2Collector
+from .collectors.ecs import ECSCollector
+from .collectors.eks import EKSCollector
+from .collectors.elasticache import ElastiCacheCollector
+from .collectors.elb import ELBCollector
 from .collectors.glue import GlueCollector
 from .collectors.iam import IAMCollector
+from .collectors.kinesis import KinesisFirehoseCollector
+from .collectors.kms import KMSCollector
+from .collectors.lakeformation import LakeFormationCollector
+from .collectors.lambda_collector import LambdaCollector
+from .collectors.misc_services import EFSCollector, StepFunctionsCollector, SystemsManagerCollector
 from .collectors.mwaa import MWAACollector
+from .collectors.rds import RDSCollector
 from .collectors.redshift import RedshiftCollector
-
-# Import collectors
+from .collectors.route53 import Route53Collector
 from .collectors.s3 import S3Collector
+from .collectors.secrets import SecretsManagerCollector
+from .collectors.security_services import SecurityServicesCollector
+from .collectors.sns_sqs import SNSSQSCollector
+
+# New collectors for comprehensive coverage
+from .collectors.vpc_costs import VPCCostsCollector
 
 
 class AWSScanner(CloudProviderScan):
@@ -166,12 +188,38 @@ class AWSScanner(CloudProviderScan):
 
     # Map of collector names to their classes for filtering
     COLLECTOR_MAP = {
+        # Original collectors
         "s3": S3Collector,
         "glue": GlueCollector,
         "athena": AthenaCollector,
         "redshift": RedshiftCollector,
         "iam": IAMCollector,
         "mwaa": MWAACollector,
+        "ec2": EC2Collector,
+        "kms": KMSCollector,
+        "rds": RDSCollector,
+        "dynamodb": DynamoDBCollector,
+        "lambda": LambdaCollector,
+        "secrets": SecretsManagerCollector,
+        "backup": BackupCollector,
+        "eks": EKSCollector,
+        "sns_sqs": SNSSQSCollector,
+        "lakeformation": LakeFormationCollector,
+        "cloudtrail": CloudTrailCollector,
+        "cloudwatch": CloudWatchLogsCollector,
+        # New collectors for comprehensive coverage
+        "vpc_costs": VPCCostsCollector,
+        "elb": ELBCollector,
+        "elasticache": ElastiCacheCollector,
+        "route53": Route53Collector,
+        "kinesis": KinesisFirehoseCollector,
+        "apigateway": APIGatewayCollector,
+        "cloudfront": CloudFrontCollector,
+        "ecs": ECSCollector,
+        "security": SecurityServicesCollector,
+        "ssm": SystemsManagerCollector,
+        "stepfunctions": StepFunctionsCollector,
+        "efs": EFSCollector,
     }
 
     @classmethod
@@ -179,6 +227,22 @@ class AWSScanner(CloudProviderScan):
         """Return list of available collector names."""
         return list(cls.COLLECTOR_MAP.keys())
 
+    # Collectors that require account_id as a parameter
+    COLLECTORS_NEEDING_ACCOUNT_ID = {
+        "vpc_costs",
+        "elb",
+        "elasticache",
+        "route53",
+        "kinesis",
+        "apigateway",
+        "cloudfront",
+        "ecs",
+        "security",
+        "ssm",
+        "stepfunctions",
+        "efs",
+    }
+
     def _initialize_collectors(self) -> list:
         """Initialize AWS service collectors based on config."""
         collectors = []
@@ -189,15 +253,22 @@ class AWSScanner(CloudProviderScan):
         # Normalize to lowercase
         requested_lower = [c.lower() for c in requested]
 
+        def create_collector(name, collector_cls):
+            """Create a collector with appropriate parameters."""
+            if name in self.COLLECTORS_NEEDING_ACCOUNT_ID:
+                return collector_cls(self.session, self.config.regions, self.config.account_id)
+            else:
+                return collector_cls(self.session, self.config.regions)
+
         # If no specific collectors requested, use all
         if not requested_lower:
-            for collector_cls in self.COLLECTOR_MAP.values():
-                collectors.append(collector_cls(self.session, self.config.regions))
+            for name, collector_cls in self.COLLECTOR_MAP.items():
+                collectors.append(create_collector(name, collector_cls))
         else:
             # Filter to only requested collectors
             for name, collector_cls in self.COLLECTOR_MAP.items():
                 if name in requested_lower:
-                    collectors.append(collector_cls(self.session, self.config.regions))
+                    collectors.append(create_collector(name, collector_cls))
 
             # Warn about unknown collectors
             known = set(self.COLLECTOR_MAP.keys())
@@ -243,18 +314,68 @@ class AWSScanner(CloudProviderScan):
 
             end_date = datetime.utcnow()
             start_date = end_date - timedelta(days=30)
-            service_costs = self.cost_explorer.get_service_costs(start_date, end_date)
+
+            # Get comprehensive cost data including Savings Plans
+            cost_data = self.cost_explorer.get_all_costs_with_savings(start_date, end_date)
+            service_costs = cost_data["service_costs"]
+            savings_plans = cost_data["savings_plans"]
             print("  → Cost data retrieved", file=sys.stderr)
+            if savings_plans.get("total_savings", 0) > 0:
+                print(
+                    f"  → Savings Plans: ${savings_plans['total_savings']:.2f} saved ({savings_plans['utilization_percent']:.1f}% utilization)",
+                    file=sys.stderr,
+                )
 
             if service_costs:
                 # Map collectors to AWS service names in Cost Explorer
                 collector_to_services = {
+                    # Original collectors
                     "s3": ["Amazon Simple Storage Service"],
                     "glue": ["AWS Glue"],
                     "athena": ["Amazon Athena"],
                     "redshift": ["Amazon Redshift"],
                     "iam": [],  # IAM is free
                     "mwaa": ["Amazon Managed Workflows for Apache Airflow"],
+                    "ec2": [
+                        "Amazon Elastic Compute Cloud - Compute",
+                        "EC2 - Other",
+                        "Amazon EC2 Container Registry (ECR)",
+                    ],
+                    "kms": ["AWS Key Management Service"],
+                    "rds": ["Amazon Relational Database Service"],
+                    "dynamodb": ["Amazon DynamoDB"],
+                    "lambda": ["AWS Lambda"],
+                    "secrets": ["AWS Secrets Manager"],
+                    "backup": ["AWS Backup"],
+                    "eks": [
+                        "Amazon Elastic Kubernetes Service",
+                        "Amazon Elastic Container Service for Kubernetes",
+                    ],
+                    "sns_sqs": [
+                        "Amazon Simple Notification Service",
+                        "Amazon Simple Queue Service",
+                    ],
+                    "lakeformation": ["AWS Lake Formation"],
+                    "cloudtrail": ["AWS CloudTrail"],
+                    "cloudwatch": ["AmazonCloudWatch", "CloudWatch Events"],
+                    # New collectors
+                    "vpc_costs": ["Amazon Virtual Private Cloud"],
+                    "elb": ["Amazon Elastic Load Balancing"],
+                    "elasticache": ["Amazon ElastiCache"],
+                    "route53": ["Amazon Route 53"],
+                    "kinesis": ["Amazon Kinesis Firehose", "Amazon Kinesis"],
+                    "apigateway": ["Amazon API Gateway"],
+                    "cloudfront": ["Amazon CloudFront"],
+                    "ecs": ["Amazon Elastic Container Service"],
+                    "security": [
+                        "Amazon GuardDuty",
+                        "Amazon Inspector",
+                        "AWS Security Hub",
+                        "AWS Config",
+                    ],
+                    "ssm": ["AWS Systems Manager"],
+                    "stepfunctions": ["AWS Step Functions"],
+                    "efs": ["Amazon Elastic File System"],
                 }
 
                 # Filter costs based on active collectors
@@ -309,6 +430,14 @@ class AWSScanner(CloudProviderScan):
                         "estimated_monthly_cost": monthly_estimate,
                         "scope": scope_note,
                         "note": "Actual costs from AWS Cost Explorer API for the last 30 days.",
+                        # Savings Plans data
+                        "savings_plans": {
+                            "total_savings": savings_plans.get("total_savings", 0),
+                            "utilization_percent": savings_plans.get("utilization_percent", 0),
+                            "coverage_percent": savings_plans.get("coverage_percent", 0),
+                            "amortized_commitment": savings_plans.get("amortized_commitment", 0),
+                            "on_demand_equivalent": savings_plans.get("on_demand_equivalent", 0),
+                        },
                     },
                 )
                 all_assets.append(cost_summary_asset)