gitflow-analytics 1.0.0__tar.gz → 1.0.1__tar.gz

gitflow_analytics-1.0.1/CHANGELOG.md

@@ -0,0 +1,94 @@
+# Changelog
+
+All notable changes to GitFlow Analytics will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [1.0.1] - 2025-07-31
+
+### Added
+- Path exclusion support for filtering boilerplate/generated files from line count metrics
+  - Configurable via `analysis.exclude.paths` in YAML configuration
+  - Default exclusions for common patterns (node_modules, lock files, minified files, etc.)
+  - Filtered metrics available as `filtered_insertions`, `filtered_deletions`, `filtered_files_changed`
+- JIRA integration for fetching story points from tickets
+  - Configurable story point field names via `jira_integration.story_point_fields`
+  - Automatic story point extraction from JIRA tickets referenced in commits
+  - Support for custom field IDs and field names
+- Organization-based repository discovery from GitHub
+  - Automatic discovery of all non-archived repositories in an organization
+  - No manual repository configuration needed for organization-wide analysis
+- Ticket platform filtering via `analysis.ticket_platforms`
+  - Ability to track only specific platforms (e.g., only JIRA, ignoring GitHub Issues)
+- Enhanced `.env` file support
+  - Automatic loading from configuration directory
+  - Validation of required environment variables
+  - Clear error messages for missing credentials
+- New CLI command: `discover-jira-fields` to find custom field IDs
+
+### Changed
+- All report generators now use filtered line counts when available
+- Cache and output directories now default to config file location (not current directory)
+- Improved developer identity resolution with better consolidation
+
+### Fixed
+- Timezone comparison errors between GitHub and local timestamps
+- License configuration in pyproject.toml for PyPI compatibility
+- Manual identity mapping format validation
+- Linting errors for better code quality
+
+### Documentation
+- Added comprehensive environment variable configuration guide
+- Complete configuration examples with `.env` and YAML files
+- Path exclusion documentation with default patterns
+- Updated README with clearer setup instructions
+
+## [1.0.0] - 2025-07-29
+
+### Added
+- Initial release of GitFlow Analytics
+- Core Git repository analysis with batch processing
+- Developer identity resolution with fuzzy matching
+- Manual identity mapping support
+- Story point extraction from commit messages
+- Multi-platform ticket tracking (GitHub, JIRA, Linear, ClickUp)
+- Comprehensive caching system with SQLite
+- CSV report generation:
+  - Weekly metrics
+  - Developer statistics
+  - Activity distribution
+  - Developer focus analysis
+  - Qualitative insights
+- Markdown narrative reports with insights
+- JSON export for API integration
+- DORA metrics calculation:
+  - Deployment frequency
+  - Lead time for changes
+  - Mean time to recovery
+  - Change failure rate
+- GitHub PR enrichment (optional)
+- Branch to project mapping
+- YAML configuration with environment variable support
+- Progress bars for long operations
+- Anonymization support for reports
+
+### Configuration Features
+- Repository definitions with project keys
+- Story point extraction patterns
+- Developer identity similarity threshold
+- Manual identity mappings
+- Default ticket platform specification
+- Branch mapping rules
+- Output format selection
+- Cache TTL configuration
+
+### Developer Experience
+- Clear CLI with helpful error messages
+- Comprehensive documentation
+- Sample configuration files
+- Progress indicators during analysis
+- Detailed logging of operations
+
+[1.0.1]: https://github.com/bobmatnyc/gitflow-analytics/releases/tag/v1.0.1
+[1.0.0]: https://github.com/bobmatnyc/gitflow-analytics/releases/tag/v1.0.0

{gitflow_analytics-1.0.0 → gitflow_analytics-1.0.1}/CLAUDE.md

@@ -42,9 +42,28 @@ The project uses SQLite for caching:
 
 Configuration uses YAML with environment variable support:
 - Variables use format: `${VARIABLE_NAME}`
+- **Environment files**: Automatically loads `.env` file from same directory as config YAML
+- **Organization support**: `github.organization` field enables automatic repository discovery
+- **Directory defaults**: Cache and reports now default to config file directory (not current working directory)
 - Default ticket platform can be specified
 - Branch mapping rules for project inference
 - Manual identity mappings for consolidating developer identities
+- Full backward compatibility with existing repository-based configurations
+
+#### Using .env Files
+
+The system automatically looks for a `.env` file in the same directory as your configuration YAML:
+```bash
+# Example .env file
+GITHUB_TOKEN=ghp_xxxxxxxxxxxxxxxxxxxx
+JIRA_ACCESS_USER=your.email@company.com
+JIRA_ACCESS_TOKEN=xxxxxxxxxxxxxxxxxxxx
+```
+
+This approach is recommended for:
+- Keeping credentials out of configuration files
+- Easy credential management across environments
+- Preventing accidental credential commits
 
 ### 5. Report Generation
 
@@ -87,6 +106,40 @@ When testing changes:
 3. Clear cache and re-run analysis
 4. Check for typos in email addresses
 
+#### Working with Organization Support
+
+1. **Organization Discovery**: When `github.organization` is specified and no repositories are manually configured:
+   - All non-archived repositories are automatically discovered from the GitHub organization
+   - Repositories are cloned to local directories if they don't exist
+   - Uses the organization name as the project key prefix if not specified
+
+2. **Testing Organization Configs**:
+   ```bash
+   # Test with organization discovery
+   gitflow-analytics analyze -c config-org.yaml --weeks 4 --validate-only
+   
+   # Run with discovered repositories
+   gitflow-analytics analyze -c config-org.yaml --weeks 4
+   ```
+
+3. **Directory Structure**: With organization support, the recommended directory structure is:
+   ```
+   /project/
+   ├── config-org.yaml       # Organization config
+   ├── repos/                # Auto-cloned repositories
+   │   ├── repo1/
+   │   ├── repo2/
+   │   └── repo3/
+   ├── .gitflow-cache/       # Cache (relative to config)
+   └── reports/              # Reports (default output location)
+   ```
+
+4. **Debugging Organization Discovery**:
+   - Check GitHub token has organization read permissions
+   - Verify organization name is correct (case-sensitive)
+   - Use `--validate-only` to test configuration without full analysis
+   - Check for API rate limiting issues
+
 ### 8. Performance Considerations
 
 - **Batch processing**: Commits are processed in batches (default: 1000)
@@ -169,6 +222,8 @@ gitflow-analytics/
 3. **Memory usage**: Large repositories can consume significant memory
 4. **Identity resolution**: Manual mappings must be applied after initial analysis
 5. **Cache invalidation**: Some changes require clearing the cache
+6. **Directory defaults**: Cache and reports now default to config file directory, not current working directory
+7. **Organization permissions**: GitHub token must have organization read access for automatic repository discovery
 
 ## Quick Commands
 

gitflow_analytics-1.0.1/PKG-INFO

@@ -0,0 +1,463 @@
+Metadata-Version: 2.4
+Name: gitflow-analytics
+Version: 1.0.1
+Summary: Analyze Git repositories for developer productivity insights
+Author-email: Bob Matyas <bobmatnyc@gmail.com>
+License: MIT
+Project-URL: Homepage, https://github.com/bobmatnyc/gitflow-analytics
+Project-URL: Documentation, https://github.com/bobmatnyc/gitflow-analytics/blob/main/README.md
+Project-URL: Repository, https://github.com/bobmatnyc/gitflow-analytics
+Project-URL: Issues, https://github.com/bobmatnyc/gitflow-analytics/issues
+Keywords: git,analytics,productivity,metrics,development
+Classifier: Development Status :: 5 - Production/Stable
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.8
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Software Development :: Version Control :: Git
+Classifier: Topic :: Software Development :: Quality Assurance
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: click>=8.1
+Requires-Dist: gitpython>=3.1
+Requires-Dist: tqdm>=4.65
+Requires-Dist: sqlalchemy>=2.0
+Requires-Dist: pandas>=2.0
+Requires-Dist: pyyaml>=6.0
+Requires-Dist: python-dateutil>=2.8
+Requires-Dist: python-dotenv>=1.0
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0; extra == "dev"
+Requires-Dist: pytest-cov>=4.0; extra == "dev"
+Requires-Dist: pytest-mock>=3.0; extra == "dev"
+Requires-Dist: ruff>=0.1.0; extra == "dev"
+Requires-Dist: mypy>=1.0; extra == "dev"
+Requires-Dist: black>=23.0; extra == "dev"
+Provides-Extra: github
+Requires-Dist: pygithub>=1.58; extra == "github"
+Provides-Extra: all
+Requires-Dist: gitflow-analytics[github]; extra == "all"
+Dynamic: license-file
+
+# GitFlow Analytics
+
+A Python package for analyzing Git repositories to generate comprehensive developer productivity reports. It extracts data directly from Git history and GitHub APIs, providing weekly summaries, productivity insights, and gap analysis.
+
+## Features
+
+- 🚀 **Multi-repository analysis** with project grouping
+- 🏢 **Organization-based repository discovery** from GitHub
+- 👥 **Developer identity resolution** and normalization
+- 📊 **Work volume analysis** (absolute vs relative effort)
+- 🎯 **Story point extraction** from commit messages and PR descriptions
+- 🎫 **Multi-platform ticket tracking** (JIRA, GitHub Issues, ClickUp, Linear)
+- 📈 **Weekly CSV reports** with productivity metrics
+- 🔒 **Data anonymization** for external sharing
+- ⚡ **Smart caching** for fast repeated analyses
+- 🔄 **Batch processing** for large repositories
+
+## Quick Start
+
+### Installation
+
+```bash
+pip install gitflow-analytics
+```
+
+### Basic Usage
+
+1. Create a configuration file (`config.yaml`):
+
+**Option A: Organization-based (Automatic Repository Discovery)**
+```yaml
+version: "1.0"
+
+github:
+  token: "${GITHUB_TOKEN}"
+  organization: "myorg"  # Automatically discovers all repositories
+
+analysis:
+  story_point_patterns:
+    - "(?:story\\s*points?|sp|pts?)\\s*[:=]\\s*(\\d+)"
+    - "\\[(\\d+)\\s*(?:sp|pts?)\\]"
+```
+
+**Option B: Repository-based (Manual Configuration)**
+```yaml
+version: "1.0"
+
+github:
+  token: "${GITHUB_TOKEN}"
+  owner: "${GITHUB_OWNER}"
+
+repositories:
+  - name: "frontend"
+    path: "~/repos/frontend"
+    github_repo: "myorg/frontend"
+    project_key: "FRONTEND"
+    
+  - name: "backend"
+    path: "~/repos/backend"
+    github_repo: "myorg/backend"
+    project_key: "BACKEND"
+
+analysis:
+  story_point_patterns:
+    - "(?:story\\s*points?|sp|pts?)\\s*[:=]\\s*(\\d+)"
+    - "\\[(\\d+)\\s*(?:sp|pts?)\\]"
+```
+
+2. Create a `.env` file in the same directory as your `config.yaml`:
+
+```bash
+# .env
+GITHUB_TOKEN=ghp_your_github_token_here
+GITHUB_OWNER=your_github_org  # Only for repository-based setup
+```
+
+3. Run the analysis:
+
+```bash
+gitflow-analytics analyze -c config.yaml
+```
+
+## Configuration Options
+
+### Environment Variables and Credentials
+
+GitFlow Analytics automatically loads environment variables from a `.env` file in the same directory as your configuration YAML. This is the recommended approach for managing credentials securely.
+
+#### Step 1: Create a `.env` file
+
+Create a `.env` file next to your configuration YAML:
+
+```bash
+# .env file (same directory as your config.yaml)
+# GitHub credentials (required)
+GITHUB_TOKEN=ghp_xxxxxxxxxxxxxxxxxxxx
+GITHUB_OWNER=myorg  # Optional: default owner for repositories
+
+# JIRA credentials (optional - only if using JIRA integration)
+JIRA_ACCESS_USER=your.email@company.com
+JIRA_ACCESS_TOKEN=xxxxxxxxxxxxxxxxxxxx
+
+# Other optional tokens
+CLICKUP_TOKEN=pk_xxxxxxxxxxxx
+LINEAR_TOKEN=lin_api_xxxxxxxxxxxx
+```
+
+#### Step 2: Reference in YAML configuration
+
+Use `${VARIABLE_NAME}` syntax in your YAML to reference environment variables:
+
+```yaml
+# config.yaml
+version: "1.0"
+
+github:
+  token: "${GITHUB_TOKEN}"        # Required
+  owner: "${GITHUB_OWNER}"        # Optional
+  organization: "${GITHUB_ORG}"   # Optional (for org-based discovery)
+
+# Optional: JIRA integration
+jira:
+  access_user: "${JIRA_ACCESS_USER}"
+  access_token: "${JIRA_ACCESS_TOKEN}"
+  base_url: "https://yourcompany.atlassian.net"
+
+# Optional: Configure which JIRA fields contain story points
+jira_integration:
+  story_point_fields:
+    - "Story Points"
+    - "customfield_10016"  # Your custom field ID
+```
+
+#### Important Notes:
+
+- **Never commit `.env` files** to version control (add to `.gitignore`)
+- If credentials are not found in the `.env` file, the tool will exit with an informative error
+- The `.env` file must be in the same directory as your YAML configuration
+- All configured services must have corresponding environment variables set
+
+### Organization vs Repository-based Setup
+
+GitFlow Analytics supports two main configuration approaches:
+
+#### Organization-based Configuration (Recommended)
+
+Automatically discovers all non-archived repositories from a GitHub organization:
+
+```yaml
+version: "1.0"
+
+github:
+  token: "${GITHUB_TOKEN}"
+  organization: "myorg"  # Your GitHub organization name
+
+# Optional: Customize analysis settings
+analysis:
+  story_point_patterns:
+    - "(?:story\\s*points?|sp|pts?)\\s*[:=]\\s*(\\d+)"
+  
+  exclude:
+    authors:
+      - "dependabot[bot]"
+      - "github-actions[bot]"
+```
+
+**Benefits:**
+- Automatically discovers new repositories as they're added to the organization
+- No need to manually configure each repository
+- Simplified configuration management
+- Perfect for teams with many repositories
+
+**Requirements:**
+- Your GitHub token must have organization read access
+- Repositories will be automatically cloned to local directories if they don't exist
+
+#### Repository-based Configuration
+
+Manually specify each repository to analyze:
+
+```yaml
+version: "1.0"
+
+github:
+  token: "${GITHUB_TOKEN}"
+  owner: "${GITHUB_OWNER}"  # Default owner for repositories
+
+repositories:
+  - name: "frontend"
+    path: "~/repos/frontend"
+    github_repo: "myorg/frontend"
+    project_key: "FRONTEND"
+    
+  - name: "backend"
+    path: "~/repos/backend"
+    github_repo: "myorg/backend"
+    project_key: "BACKEND"
+
+analysis:
+  story_point_patterns:
+    - "(?:story\\s*points?|sp|pts?)\\s*[:=]\\s*(\\d+)"
+```
+
+**Benefits:**
+- Fine-grained control over which repositories to analyze
+- Custom project keys and local paths
+- Works with mixed-ownership repositories
+- Compatible with existing configurations
+
+### Directory Defaults
+
+GitFlow Analytics now defaults cache and report directories to be relative to the configuration file location:
+
+- **Reports**: Default to same directory as config file (unless overridden with `--output`)
+- **Cache**: Default to `.gitflow-cache/` in config file directory
+- **Backward compatibility**: Absolute paths in configuration continue to work as before
+
+Example directory structure:
+```
+/project/
+├── config.yaml          # Configuration file
+├── weekly_metrics.csv    # Reports generated here by default
+├── summary.csv
+└── .gitflow-cache/       # Cache directory
+    ├── gitflow_cache.db
+    └── identities.db
+```
+
+## Command Line Interface
+
+### Main Commands
+
+```bash
+# Analyze repositories
+gitflow-analytics analyze -c config.yaml --weeks 12 --output ./reports
+
+# Show cache statistics
+gitflow-analytics cache-stats -c config.yaml
+
+# List known developers
+gitflow-analytics list-developers -c config.yaml
+
+# Merge developer identities
+gitflow-analytics merge-identity -c config.yaml dev1_id dev2_id
+
+# Discover JIRA story point fields
+gitflow-analytics discover-jira-fields -c config.yaml
+```
+
+### Options
+
+- `--weeks, -w`: Number of weeks to analyze (default: 12)
+- `--output, -o`: Output directory for reports (default: ./reports)
+- `--anonymize`: Anonymize developer information
+- `--no-cache`: Disable caching for fresh analysis
+- `--clear-cache`: Clear cache before analysis
+- `--validate-only`: Validate configuration without running
+
+## Complete Configuration Example
+
+Here's a complete example showing `.env` file and corresponding YAML configuration:
+
+### `.env` file
+```bash
+# GitHub Configuration
+GITHUB_TOKEN=ghp_xxxxxxxxxxxxxxxxxxxx
+GITHUB_ORG=EWTN-Global
+
+# JIRA Configuration
+JIRA_ACCESS_USER=developer@ewtn.com
+JIRA_ACCESS_TOKEN=ATATT3xxxxxxxxxxx
+
+# Optional: Other integrations
+# CLICKUP_TOKEN=pk_xxxxxxxxxxxx
+# LINEAR_TOKEN=lin_api_xxxxxxxxxxxx
+```
+
+### `config.yaml` file
+```yaml
+version: "1.0"
+
+# GitHub configuration with organization discovery
+github:
+  token: "${GITHUB_TOKEN}"
+  organization: "${GITHUB_ORG}"
+
+# JIRA integration for story points
+jira:
+  access_user: "${JIRA_ACCESS_USER}"
+  access_token: "${JIRA_ACCESS_TOKEN}"
+  base_url: "https://ewtn.atlassian.net"
+
+jira_integration:
+  enabled: true
+  fetch_story_points: true
+  story_point_fields:
+    - "Story point estimate"     # Your field name
+    - "customfield_10016"        # Fallback field ID
+
+# Analysis configuration
+analysis:
+  # Only track JIRA tickets (ignore GitHub issues, etc.)
+  ticket_platforms:
+    - jira
+  
+  # Exclude bot commits and boilerplate files
+  exclude:
+    authors:
+      - "dependabot[bot]"
+      - "renovate[bot]"
+    paths:
+      - "**/node_modules/**"
+      - "**/*.min.js"
+      - "**/package-lock.json"
+  
+  # Developer identity consolidation
+  identity:
+    similarity_threshold: 0.85
+    manual_mappings:
+      - primary_email: "john.doe@company.com"
+        aliases:
+          - "jdoe@oldcompany.com"
+          - "john@personal.com"
+
+# Output configuration
+output:
+  directory: "./reports"
+  formats:
+    - csv
+    - markdown
+```
+
+## Output Reports
+
+The tool generates three CSV reports:
+
+1. **Weekly Metrics** (`weekly_metrics_YYYYMMDD.csv`)
+   - Week-by-week developer productivity
+   - Story points, commits, lines changed
+   - Ticket coverage percentages
+   - Per-project breakdown
+
+2. **Summary Statistics** (`summary_YYYYMMDD.csv`)
+   - Overall project statistics
+   - Platform-specific ticket counts
+   - Top contributors
+
+3. **Developer Report** (`developers_YYYYMMDD.csv`)
+   - Complete developer profiles
+   - Total contributions
+   - Identity aliases
+
+## Story Point Patterns
+
+Configure custom regex patterns to match your team's story point format:
+
+```yaml
+story_point_patterns:
+  - "SP: (\\d+)"           # SP: 5
+  - "\\[([0-9]+) pts\\]"   # [3 pts]
+  - "estimate: (\\d+)"     # estimate: 8
+```
+
+## Ticket Platform Support
+
+Automatically detects and tracks tickets from:
+- **JIRA**: `PROJ-123`
+- **GitHub**: `#123`, `GH-123`
+- **ClickUp**: `CU-abc123`
+- **Linear**: `ENG-123`
+
+### JIRA Integration
+
+GitFlow Analytics can fetch story points directly from JIRA tickets. Configure your JIRA instance:
+
+```yaml
+jira:
+  access_user: "${JIRA_ACCESS_USER}"
+  access_token: "${JIRA_ACCESS_TOKEN}"
+  base_url: "https://your-company.atlassian.net"
+
+jira_integration:
+  enabled: true
+  story_point_fields:
+    - "Story point estimate"  # Your custom field name
+    - "customfield_10016"     # Or use field ID
+```
+
+To discover your JIRA story point fields:
+```bash
+gitflow-analytics discover-jira-fields -c config.yaml
+```
+
+## Caching
+
+The tool uses SQLite for intelligent caching:
+- Commit analysis results
+- Developer identity mappings
+- Pull request data
+
+Cache is automatically managed with configurable TTL.
+
+## Developer Identity Resolution
+
+Intelligently merges developer identities across:
+- Different email addresses
+- Name variations
+- GitHub usernames
+
+Manual overrides supported in configuration.
+
+## Contributing
+
+Contributions are welcome! Please feel free to submit a Pull Request.
+
+## License
+
+This project is licensed under the MIT License - see the LICENSE file for details.