aws-resource-inventory 0.1.0__tar.gz

aws_resource_inventory-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 Ishan Sharma
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

aws_resource_inventory-0.1.0/PKG-INFO

@@ -0,0 +1,557 @@
+Metadata-Version: 2.4
+Name: aws-resource-inventory
+Version: 0.1.0
+Summary: AWS multi-service resource inventory with tag-based filtering and advanced optimizations
+License-Expression: MIT
+License-File: LICENSE
+Keywords: aws,scanner,cloud,infrastructure,devops
+Author: Ishan Sharma
+Requires-Python: >=3.10
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: System Administrators
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: System :: Systems Administration
+Classifier: Topic :: Internet :: WWW/HTTP
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Requires-Dist: boto3 (>=1.40.34)
+Requires-Dist: botocore (>=1.40.34)
+Requires-Dist: click (>=8.3.0)
+Requires-Dist: pyfiglet (>=1.0.4)
+Requires-Dist: rich (>=14.1.0)
+Requires-Dist: setuptools (>=80.9.0)
+Requires-Dist: typer (>=0.17.4)
+Requires-Dist: urllib3 (>=2.5.0)
+Project-URL: Homepage, https://github.com/ishuar/aws-resource-inventory
+Project-URL: Repository, https://github.com/ishuar/aws-resource-inventory
+Description-Content-Type: text/markdown
+
+# AWS Resource Inventory
+
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+[![Python 3.10+](https://img.shields.io/badge/python-3.10+-blue.svg)](https://www.python.org/downloads/)
+[![Code Style: Black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
+[![pre-commit.ci status](https://results.pre-commit.ci/badge/github/ishuar/aws-resource-inventory/main.svg)](https://results.pre-commit.ci/latest/github/ishuar/aws-resource-inventory/main)
+
+A comprehensive AWS multi-service scanner with tag-based filtering, parallel processing, advanced logging capabilities, and optimization features. This tool enables efficient discovery and analysis of AWS resources across multiple regions and services with intelligent caching, rich output formats, and detailed AWS API tracing.
+
+## Features
+
+- Scans eight AWS services — EC2, S3, ECS, EFS, VPC, RDS, ELB, and Auto Scaling — across multiple regions concurrently, with or without tag filters.
+- Discovers resources from 100+ AWS services via the Resource Groups Tagging API (`--all-services`, requires a tag filter).
+- Table, JSON, and Markdown output; results are rendered in the terminal and written to a file for further processing.
+- Result caching with a 10-minute TTL (`--cache` / `--no-cache`).
+- Configurable parallelism per region (`--max-workers`) and per service (`--service-workers`), dry-run preview, continuous refresh mode, and graceful Ctrl+C handling.
+- Debug traces (`--debug`), full AWS API tracing (`--verbose`), and custom log files (`--log-file`) — see [Logging Architecture](docs/LOGGING_ARCHITECTURE.md).
+
+## 🏗️ Supported AWS Services
+
+| Service          | Description                                                                                                             | Resources Scanned                                                 |
+|------------------|-------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------|
+| **All Services** | With [Resource Groups Tagging API](https://docs.aws.amazon.com/resourcegroupstagging/latest/APIReference/overview.html) | 100+ AWS services when using tags                                 |
+| **EC2**          | Elastic Compute Cloud                                                                                                   | Instances, Volumes, Security Groups, AMIs, Snapshots              |
+| **S3**           | Simple Storage Service                                                                                                  | Buckets and their configurations                                  |
+| **ECS**          | Elastic Container Service                                                                                               | Clusters, Services, Task Definitions, Capacity Providers          |
+| **VPC**          | Virtual Private Cloud                                                                                                   | VPCs, Subnets, Route Tables, IGWs, NAT Gateways, DHCP Options, Peering Connections, Endpoints |
+| **Auto Scaling** | Auto Scaling Groups                                                                                                     | ASGs, Launch Configurations, Launch Templates                     |
+| **ELB**          | Elastic Load Balancing                                                                                                  | Load Balancers (ALB/NLB), Target Groups, Listeners, Listener Rules |
+| **RDS**          | Relational Database Service                                                                                             | DB Instances, DB Clusters, DB Snapshots, Aurora Cluster Snapshots |
+| **EFS**          | Elastic File System                                                                                                     | File Systems (with size and mount-target details)                 |
+
+> **📚 Architecture Details**: For detailed information about the scanning architecture and service implementation patterns, see [Architecture Documentation](docs/Architecture.md).
+
+## 📋 Prerequisites
+
+Before installing the AWS Resource Inventory, ensure you have the following dependencies:
+
+### Required Software
+- **Python 3.10+** - Runtime environment
+- **Poetry** - Python dependency management and packaging
+- **pip** - Python package installer
+- **AWS CLI** - AWS command line interface for authentication
+- **pre-commit** - Git hooks framework (for development)
+
+### AWS Configuration
+- Valid AWS credentials configured via:
+  - AWS SSO (`aws sso login`)
+  - AWS CLI (`aws configure`)
+  - Environment variables
+  - IAM roles (for EC2/containers)
+
+### Required IAM Permissions
+
+The scanner is read-only. This policy covers every API call it makes —
+the eight service scanners, the credential check (`sts`), and the tag
+scan (`tag:GetResources`, only exercised with `--tag-key`/`--tag-value`/
+`--all-services`):
+
+```json
+{
+    "Version": "2012-10-17",
+    "Statement": [
+        {
+            "Sid": "AwsResourceInventoryReadOnly",
+            "Effect": "Allow",
+            "Action": [
+                "sts:GetCallerIdentity",
+                "ec2:Describe*",
+                "s3:ListAllMyBuckets",
+                "s3:GetBucketLocation",
+                "s3:GetBucketTagging",
+                "ecs:List*",
+                "ecs:Describe*",
+                "elasticloadbalancing:Describe*",
+                "elasticfilesystem:Describe*",
+                "rds:Describe*",
+                "autoscaling:Describe*",
+                "tag:GetResources"
+            ],
+            "Resource": "*"
+        }
+    ]
+}
+```
+
+> [!NOTE]
+> `ec2:Describe*` also covers the VPC scanner and Auto Scaling launch
+> templates — both use the EC2 API. The AWS-managed `ReadOnlyAccess`
+> policy works too if you prefer not to maintain a custom one.
+
+## 🚀 **Installation**
+
+### Automated Setup (Recommended)
+
+The easiest way to get started is using our automated setup script:
+
+```bash
+# Clone the repository
+git clone https://github.com/ishuar/aws-resource-inventory.git
+cd aws-resource-inventory
+
+# Run the automated setup script
+./setup.sh
+```
+
+The setup script will:
+1. ✅ Check and install Python 3.10+
+2. ✅ Install Poetry (Python dependency manager)
+3. ✅ Install pre-commit (Git hooks framework)
+4. ✅ Set up pre-commit hooks
+5. ✅ Install all project dependencies via Poetry
+6. ✅ Run verification tests
+7. ✅ Install AWS CLI (if not present)
+8. ✅ Provide AWS configuration guidance
+
+> [!Tip]
+> For detailed setup instructions and troubleshooting, see [setup.sh](setup.sh).
+
+### Manual Installation
+
+If you prefer manual installation:
+
+```bash
+# Install dependencies (macOS with Homebrew)
+brew install python3 poetry pre-commit awscli
+
+# Install project dependencies
+poetry install
+
+# Set up pre-commit hooks
+pre-commit install --install-hooks
+
+# Verify installation
+./run_quick_tests.sh
+```
+
+## 💻 Usage
+
+### Command Structure
+
+All commands follow this pattern:
+```bash
+poetry run aws-inventory [GLOBAL OPTIONS] COMMAND [COMMAND OPTIONS]
+```
+
+Global options apply to all commands: `--verbose` / `-v` (AWS API tracing,
+use with `--debug`) and `--log-file` / `-l` (custom log file path). The
+main command is `scan`.
+
+> [!IMPORTANT]
+> Global options must come **before** the command. `aws-inventory scan --verbose`
+> fails with `No such option: --verbose` — the correct form is
+> `aws-inventory --verbose scan`.
+
+### Basic Commands
+
+```bash
+# Display help and available commands
+poetry run aws-inventory --help
+
+# Display scan command help and options
+poetry run aws-inventory scan --help
+
+# Basic scan with default settings (all supported services)
+poetry run aws-inventory scan --regions us-east-1,eu-west-1,eu-central-1,us-west-2
+
+# Scan specific services
+poetry run aws-inventory scan --service ec2
+
+# Scan specific regions
+poetry run aws-inventory scan --regions us-east-1,eu-west-1
+```
+
+### Debug and Logging Options
+
+```bash
+# Enable debug mode for detailed execution traces
+poetry run aws-inventory scan --debug --regions us-east-1
+
+# Enable verbose AWS API tracing (requires --debug)
+poetry run aws-inventory --verbose scan --debug --service ec2
+
+# Custom log file for debug output
+poetry run aws-inventory --log-file /tmp/my-scan.log scan --debug --regions us-east-1
+
+# Combine verbose logging with custom log file
+poetry run aws-inventory --verbose --log-file /tmp/aws-api-trace.log scan --debug --service ec2 --service s3
+
+# Debug with dry run (no actual scanning)
+poetry run aws-inventory --verbose scan --debug --dry-run --service vpc
+```
+
+### Service-Specific Scanning
+
+```bash
+# Scan only EC2 resources
+poetry run aws-inventory scan --service ec2
+
+# Scan multiple services
+poetry run aws-inventory scan --service ec2 --service s3 --service vpc
+
+# Scan all built-in services in specific regions
+poetry run aws-inventory scan --regions us-east-1,us-west-2
+
+# Combine service and region filtering
+poetry run aws-inventory scan --service ec2 --regions eu-central-1,eu-west-1
+
+# Scan ALL AWS services using Resource Groups API (requires tags)
+poetry run aws-inventory scan --all-services --tag-key Environment --tag-value Production
+```
+
+### Tag-Based Filtering
+
+```bash
+# Filter by environment tag
+poetry run aws-inventory scan --tag-key Environment --tag-value Production
+
+# Filter by application tag
+poetry run aws-inventory scan --tag-key app --tag-value web-server
+
+# Filter by cost center in specific regions
+poetry run aws-inventory scan --regions us-east-1 --tag-key CostCenter --tag-value Engineering
+```
+
+### Output Formats
+
+```bash
+# Default table format (human-readable)
+poetry run aws-inventory scan --format table
+
+# JSON format for programmatic processing
+poetry run aws-inventory scan --format json --output results.json
+
+# Markdown format for documentation
+poetry run aws-inventory scan --format md --output report.md
+
+# Export filtered results to JSON
+poetry run aws-inventory scan --tag-key Environment --tag-value Production --format json --output prod-resources.json
+```
+
+### Advanced Options
+
+```bash
+# Dry run (preview without execution)
+poetry run aws-inventory scan --dry-run --service ec2
+
+# Disable caching for fresh data
+poetry run aws-inventory scan --no-cache
+
+# Configure worker threads for performance
+poetry run aws-inventory scan --max-workers 10 --service-workers 6
+
+# Continuous refresh mode with custom interval
+poetry run aws-inventory scan --refresh --refresh-interval 30 --service ec2
+
+# Debug mode with performance timing
+poetry run aws-inventory --verbose scan --debug --max-workers 1 --service ec2
+```
+
+### Real-World Examples
+
+```bash
+# Production infrastructure audit with comprehensive logging
+poetry run aws-inventory --verbose --log-file prod-audit.log scan \
+    --debug --tag-key Environment --tag-value Production \
+    --format json --output production-audit.json
+
+# Regional compliance check with detailed tracing
+poetry run aws-inventory --verbose --log-file compliance-trace.log scan \
+    --debug --regions eu-west-1,eu-central-1 \
+    --service ec2 --format md --output eu-compliance-report.md
+
+# Application-specific resource discovery across all AWS services
+poetry run aws-inventory scan \
+    --all-services --tag-key Application --tag-value MyApp \
+    --format table --regions us-east-1
+
+# Development environment troubleshooting with verbose logging
+poetry run aws-inventory --verbose --log-file dev-debug.log scan \
+    --debug --regions us-west-2 \
+    --tag-key Environment --tag-value Development \
+    --no-cache --dry-run
+
+# Performance analysis with sequential processing
+poetry run aws-inventory --verbose --log-file perf-analysis.log scan \
+    --debug --max-workers 1 --service-workers 1 \
+    --service ec2 --service s3 --regions us-east-1
+```
+
+## 🔧 Configuration
+
+### AWS Profile Configuration
+
+```bash
+# Set AWS profile
+export AWS_PROFILE=your-profile-name
+
+# Login via SSO
+aws sso login --profile $AWS_PROFILE
+
+# Verify credentials
+aws sts get-caller-identity
+```
+
+### Logging Configuration
+
+The scanner features a comprehensive logging system with multiple configuration options:
+
+```bash
+# Global logging options (apply to all commands)
+poetry run aws-inventory --verbose --log-file /path/to/logfile.log scan --debug
+
+# Debug modes explained:
+# --debug: Enable debug mode with rich console output and file logging
+# --verbose: Enable AWS API tracing (requires --debug)
+# --log-file: Custom log file path (default: .debug_logs/aws_scanner_debug_TIMESTAMP.log)
+```
+
+Logging levels: normal runs show progress and results; `--debug` adds
+execution traces, timing, and caller context; `--verbose` together with
+`--debug` adds full AWS API tracing including HTTP requests and responses.
+
+> [!Tip]
+> **📖 Detailed Logging Guide**: For comprehensive logging documentation, configuration examples, and troubleshooting, see [Logging Architecture](docs/LOGGING_ARCHITECTURE.md).
+
+### Performance Tuning
+
+- `--max-workers` (1-20) controls region-level parallelism; `--service-workers` (1-10) controls service-level parallelism.
+- `--cache` speeds up repeated scans (10-minute TTL); `--no-cache` forces fresh data.
+
+> [!NOTE]
+> Verbose logging adds roughly 10-20% overhead — use it for troubleshooting, not routine scans.
+
+## 📁 **Project Structure**
+
+```
+aws-resource-inventory/
+├── aws_scanner.py               # Scan orchestration across regions
+├── cli.py                       # Command-line interface (typer)
+├── setup.sh                     # Automated setup script
+├── run_quick_tests.sh           # Smoke-test script
+├── pyproject.toml               # Project configuration and dependencies
+│
+├── services/                    # One scanner module per AWS service
+│   ├── registry.py              # Single source of truth: service → scanner + output processor
+│   ├── ec2_service.py           # Instances, volumes, security groups, AMIs, snapshots
+│   ├── s3_service.py            # Buckets (region-filtered, tag-enriched)
+│   ├── ecs_service.py           # Clusters, services, task definitions, capacity providers
+│   ├── efs_service.py           # File systems
+│   ├── elb_service.py           # Load balancers, target groups, listeners, rules
+│   ├── rds_service.py           # DB instances, clusters, snapshots (incl. Aurora)
+│   ├── vpc_service.py           # VPC networking components
+│   └── autoscaling_service.py   # ASGs, launch configurations, launch templates
+│
+├── aws_scanner_lib/             # Core library modules
+│   ├── engine.py                # Shared scanning engine: pagination, concurrency, error policy
+│   ├── records.py               # Resource — the typed record every output consumes
+│   ├── clients.py               # The only boto3 client factory (pooling, adaptive retries)
+│   ├── scan.py                  # Region/service scan orchestration + caching hooks
+│   ├── outputs.py               # Table / JSON / Markdown output processing
+│   ├── resource_groups_utils.py # Resource Groups Tagging API path (--all-services, tags)
+│   ├── cache.py                 # Result caching with TTL
+│   └── logging.py               # Unified logging with AWS API tracing
+│
+├── scripts/
+│   └── e2e-diff.sh              # Before/after functional comparison against real AWS
+│
+├── docs/                        # Documentation
+│   ├── adr/                     # Architecture decision records
+│   ├── Architecture.md          # System architecture and design patterns
+│   └── LOGGING_ARCHITECTURE.md  # Logging system documentation
+│
+└── tests/                       # Test suite — runs with ZERO AWS credentials (moto)
+```
+
+## 🧪 Testing
+
+> [!NOTE]
+> The suite needs no AWS credentials: moto fakes AWS, and the test fixtures
+> force fake credentials so no test can ever touch a real account.
+
+```bash
+# Run the full suite (coverage is reported automatically)
+poetry run pytest
+
+# Run one test file
+poetry run pytest tests/test_engine.py
+
+# Quick smoke checks (help, dry-run, formats)
+./run_quick_tests.sh
+```
+
+After merging a change that touches scan behaviour, verify against real
+AWS with the before/after comparison script:
+
+```bash
+scripts/e2e-diff.sh                        # compares origin/main~1 vs origin/main
+scripts/e2e-diff.sh --tag-key team         # exercise the tag-scan path too
+```
+
+## 🐛 Troubleshooting
+
+### Common Issues
+
+1. **AWS Credentials**: Ensure AWS credentials are properly configured
+   ```bash
+   aws sts get-caller-identity
+   ```
+
+2. **Python Version**: Verify Python 3.10+ is installed
+   ```bash
+   python3 --version
+   ```
+
+3. **Dependencies**: Reinstall dependencies if needed
+   ```bash
+   poetry install --no-cache
+   ```
+
+4. **Permissions**: Ensure your AWS user/role has necessary permissions for the services you're scanning
+
+### Debug and Logging Troubleshooting
+
+The advanced logging system provides powerful debugging capabilities:
+
+```bash
+# Basic debug information
+poetry run aws-inventory scan --debug --dry-run
+
+# Verbose AWS API tracing for troubleshooting
+poetry run aws-inventory --verbose --log-file debug-trace.log scan --debug --service ec2
+
+# Check debug log files (automatically created)
+ls .debug_logs/
+
+# Monitor real-time logging
+tail -f .debug_logs/aws_scanner_debug_*.log
+
+# Filter AWS API calls
+grep -E "(boto|botocore|HTTP)" .debug_logs/aws_scanner_debug_*.log
+```
+
+Common logging issues: no AWS API logs usually means one of `--debug` or
+`--verbose` is missing (both are required, and `--verbose` goes before
+`scan`); for sequential, readable debugging use `--max-workers 1`; if log
+files are missing, check permissions on the `.debug_logs/` directory.
+
+> [!Tip]
+> **🔧 Advanced Troubleshooting**: For detailed logging troubleshooting and configuration options, see [Logging Architecture - Troubleshooting Section](docs/LOGGING_ARCHITECTURE.md#troubleshooting).
+
+## 📌 Quick Reference
+
+### Most Common Commands
+
+```bash
+# Quick scan with basic output
+poetry run aws-inventory scan --regions us-east-1
+
+# Debug mode with detailed logging
+poetry run aws-inventory scan --debug --regions us-east-1
+
+# Full AWS API tracing (development/troubleshooting)
+poetry run aws-inventory --verbose --log-file trace.log scan --debug --service ec2
+
+# Tag-based filtering across all AWS services
+poetry run aws-inventory scan --all-services --tag-key Environment --tag-value Production
+
+# Production audit with comprehensive logging
+poetry run aws-inventory --verbose --log-file audit.log scan --debug \
+    --tag-key Environment --tag-value Production --format json --output audit.json
+```
+
+### Flag Combinations Guide
+
+| Scenario         | Command Pattern                                          | Purpose                        |
+|------------------|----------------------------------------------------------|--------------------------------|
+| **Basic Scan**   | `poetry run aws-inventory scan`                            | Standard resource discovery    |
+| **Debug Mode**   | `poetry run aws-inventory scan --debug`                    | Detailed execution information |
+| **API Tracing**  | `poetry run aws-inventory --verbose scan --debug`          | Full AWS API call logging      |
+| **Custom Logs**  | `poetry run aws-inventory --log-file path scan --debug`    | Custom log file location       |
+| **All Services** | `poetry run aws-inventory scan --all-services --tag-key X` | Discover 100+ AWS services     |
+
+## 📖 Documentation
+
+The project includes comprehensive documentation covering all aspects of the system:
+
+- [Architecture Documentation](docs/Architecture.md) — system design, component interactions, and patterns
+- [Architecture Decision Records](docs/adr/) — why things are built the way they are
+- [Shell Completion](docs/SHELL_COMPLETION.md) — tab-completion setup for the CLI
+- [Logging Architecture](docs/LOGGING_ARCHITECTURE.md) — the logging system: API tracing, configuration, integration patterns, troubleshooting, performance
+
+## 🤝 Contributing
+
+1. Fork the repository
+2. Create a feature branch: `git checkout -b feature/amazing-feature`
+3. Install development dependencies: Use [`./setup.sh`](./setup.sh)
+4. Set up pre-commit hooks: `pre-commit install --install-hooks`
+5. Make your changes and run tests: `poetry run pytest`
+6. Test logging changes: `poetry run aws-inventory --verbose scan --debug --dry-run`
+7. Commit with a conventional-commit message (titles feed the release changelog): `git commit -m 'feat: add amazing feature'`
+8. Push to the branch: `git push origin feature/amazing-feature`
+9. Open a Pull Request
+
+> [!TIP]
+> **Development Notes:**
+> - Use the debug and verbose flags extensively during development
+> - Check the logging architecture documentation when modifying logging behavior
+> - Ensure all new features include appropriate logging and error handling
+
+## 📝 License
+
+This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
+
+## 🙏 Acknowledgments
+
+- Built with [Typer](https://typer.tiangolo.com/) for CLI interface
+- Styled with [Rich](https://rich.readthedocs.io/) for beautiful console output
+- Powered by [Boto3](https://boto3.amazonaws.com/) for AWS integration
+- Managed with [Poetry](https://python-poetry.org/) for dependency management
+
+---
+
+> _Made with ❤️ for the AWS community_
+