iocparser-tool 2.0.0__tar.gz → 3.0.0__tar.gz

iocparser_tool-3.0.0/CONTRIBUTING.md

@@ -0,0 +1,363 @@
+# Contributing to IOCParser
+
+Thank you for your interest in contributing to IOCParser! This document provides guidelines and instructions for contributing to the project.
+
+## Table of Contents
+
+- [Code of Conduct](#code-of-conduct)
+- [Getting Started](#getting-started)
+- [Development Setup](#development-setup)
+- [Making Contributions](#making-contributions)
+- [Coding Standards](#coding-standards)
+- [Testing](#testing)
+- [Documentation](#documentation)
+- [Submitting Changes](#submitting-changes)
+
+## Code of Conduct
+
+By participating in this project, you agree to abide by our code of conduct:
+
+- Be respectful and inclusive
+- Welcome newcomers and help them get started
+- Focus on constructive criticism
+- Accept feedback gracefully
+- Prioritize the project's best interests
+
+## Getting Started
+
+1. Fork the repository on GitHub
+2. Clone your fork locally:
+   ```bash
+   git clone https://github.com/YOUR_USERNAME/iocparser.git
+   cd iocparser
+   ```
+3. Add the upstream repository:
+   ```bash
+   git remote add upstream https://github.com/seifreed/iocparser.git
+   ```
+
+## Development Setup
+
+### Prerequisites
+
+- Python 3.8 or higher
+- pip and virtualenv
+- Git
+
+### Setting Up Your Environment
+
+1. Create a virtual environment:
+   ```bash
+   python -m venv venv
+   source venv/bin/activate  # On Windows: venv\Scripts\activate
+   ```
+
+2. Install the package in development mode:
+   ```bash
+   pip install -e .
+   ```
+
+3. Install development dependencies:
+   ```bash
+   pip install -e ".[dev]"
+   ```
+
+4. Install pre-commit hooks:
+   ```bash
+   pre-commit install
+   ```
+
+## Making Contributions
+
+### Types of Contributions
+
+We welcome various types of contributions:
+
+- **Bug Fixes**: Fix issues reported in GitHub Issues
+- **New Features**: Add new IOC extraction capabilities
+- **Performance Improvements**: Optimize existing code
+- **Documentation**: Improve or add documentation
+- **Tests**: Add or improve test coverage
+- **Refactoring**: Improve code structure and readability
+
+### Workflow
+
+1. **Create a Branch**: 
+   ```bash
+   git checkout -b feature/your-feature-name
+   # or
+   git checkout -b fix/issue-number
+   ```
+
+2. **Make Your Changes**: Follow our coding standards (see below)
+
+3. **Write/Update Tests**: Ensure your changes are tested
+
+4. **Run Tests Locally**:
+   ```bash
+   pytest tests/
+   ```
+
+5. **Check Code Quality**:
+   ```bash
+   black iocparser/
+   flake8 iocparser/
+   mypy iocparser/
+   ```
+
+6. **Commit Your Changes**:
+   ```bash
+   git add .
+   git commit -m "feat: add support for extracting X IOC type"
+   ```
+
+### Commit Message Format
+
+We use conventional commits format:
+
+- `feat:` New feature
+- `fix:` Bug fix
+- `docs:` Documentation changes
+- `style:` Code style changes (formatting, etc.)
+- `refactor:` Code refactoring
+- `perf:` Performance improvements
+- `test:` Test additions or changes
+- `chore:` Maintenance tasks
+
+Examples:
+```
+feat: add extraction support for MITRE ATT&CK IDs
+fix: correct IPv6 regex pattern matching
+docs: update README with new IOC types
+perf: optimize domain extraction for large files
+```
+
+## Coding Standards
+
+### Python Style Guide
+
+We follow PEP 8 with some modifications:
+
+- Maximum line length: 100 characters
+- Use type hints for all function signatures
+- Use docstrings for all public functions and classes
+
+### Code Formatting
+
+We use `black` for automatic code formatting:
+
+```bash
+black --line-length 100 iocparser/
+```
+
+### Type Hints
+
+All new code should include type hints:
+
+```python
+from typing import List, Dict, Optional
+
+def extract_iocs(text: str, defang: bool = True) -> Dict[str, List[str]]:
+    """
+    Extract IOCs from text.
+    
+    Args:
+        text: Input text to process
+        defang: Whether to defang extracted IOCs
+        
+    Returns:
+        Dictionary mapping IOC types to lists of IOCs
+    """
+    pass
+```
+
+### Docstrings
+
+Use Google-style docstrings:
+
+```python
+def function_name(param1: str, param2: int) -> bool:
+    """
+    Brief description of the function.
+    
+    More detailed description if needed.
+    
+    Args:
+        param1: Description of param1
+        param2: Description of param2
+        
+    Returns:
+        Description of return value
+        
+    Raises:
+        ValueError: When invalid input is provided
+    """
+    pass
+```
+
+## Testing
+
+### Writing Tests
+
+- Write tests for all new functionality
+- Place tests in the `tests/` directory
+- Use descriptive test names
+- Include edge cases and error conditions
+
+Example test:
+
+```python
+def test_extract_md5_valid():
+    """Test extraction of valid MD5 hashes."""
+    extractor = IOCExtractor()
+    text = "Hash: 5f4dcc3b5aa765d61d8327deb882cf99"
+    result = extractor.extract_md5(text)
+    assert len(result) == 1
+    assert "5f4dcc3b5aa765d61d8327deb882cf99" in result
+```
+
+### Running Tests
+
+```bash
+# Run all tests
+pytest tests/
+
+# Run with coverage
+pytest tests/ --cov=iocparser --cov-report=term-missing
+
+# Run specific test file
+pytest tests/test_extractors.py
+
+# Run tests in verbose mode
+pytest tests/ -v
+```
+
+### Performance Testing
+
+For performance-critical changes:
+
+```bash
+pytest tests/test_performance.py --benchmark-only
+```
+
+## Documentation
+
+### Code Documentation
+
+- Add docstrings to all public functions and classes
+- Update existing docstrings when changing functionality
+- Include examples in docstrings for complex functions
+
+### README Updates
+
+Update the README.md when:
+- Adding new IOC types
+- Changing command-line interface
+- Adding new features
+- Changing installation requirements
+
+### API Documentation
+
+For significant API changes, update the documentation:
+
+```bash
+# Generate API documentation
+sphinx-build -b html docs/ docs/_build/
+```
+
+## Submitting Changes
+
+### Pull Request Process
+
+1. **Update Your Fork**:
+   ```bash
+   git fetch upstream
+   git checkout main
+   git merge upstream/main
+   ```
+
+2. **Rebase Your Branch**:
+   ```bash
+   git checkout your-branch
+   git rebase main
+   ```
+
+3. **Push to Your Fork**:
+   ```bash
+   git push origin your-branch
+   ```
+
+4. **Create Pull Request**:
+   - Go to GitHub and create a pull request
+   - Fill in the PR template
+   - Link related issues
+   - Ensure CI checks pass
+
+### PR Requirements
+
+Before submitting a PR, ensure:
+
+- [ ] All tests pass
+- [ ] Code follows style guidelines
+- [ ] Documentation is updated
+- [ ] Commit messages follow convention
+- [ ] PR description clearly explains changes
+- [ ] Related issues are linked
+
+### Review Process
+
+1. Maintainers will review your PR
+2. Address any feedback or requested changes
+3. Once approved, your PR will be merged
+
+## Adding New IOC Types
+
+When adding support for new IOC types:
+
+1. **Add Regex Pattern** in `extractor.py`:
+   ```python
+   self.patterns['new_ioc_type'] = re.compile(r'pattern')
+   ```
+
+2. **Add Extraction Method**:
+   ```python
+   def extract_new_ioc_type(self, text: str) -> List[str]:
+       """Extract new IOC type from text."""
+       return self._extract_pattern(text, 'new_ioc_type')
+   ```
+
+3. **Update `extract_all` Method**:
+   ```python
+   ('new_ioc_type', self.extract_new_ioc_type),
+   ```
+
+4. **Add Tests**:
+   ```python
+   def test_extract_new_ioc_type():
+       # Test implementation
+   ```
+
+5. **Update Documentation**:
+   - Add to README.md features list
+   - Update API documentation
+
+## Questions and Support
+
+If you have questions:
+
+1. Check existing [issues](https://github.com/seifreed/iocparser/issues)
+2. Search the [documentation](https://github.com/seifreed/iocparser#readme)
+3. Create a new issue with the question label
+
+## License
+
+By contributing to IOCParser, you agree that your contributions will be licensed under the MIT License.
+
+## Recognition
+
+Contributors will be recognized in:
+- The project's contributors list
+- Release notes for significant contributions
+- The AUTHORS file (for major contributors)
+
+Thank you for contributing to IOCParser!

iocparser_tool-3.0.0/Makefile

@@ -0,0 +1,169 @@
+.PHONY: help install install-dev clean test lint format security check-all pre-commit build docs
+
+# Variables
+PYTHON := python3
+PIP := $(PYTHON) -m pip
+PROJECT_NAME := iocparser
+SRC_DIR := iocparser
+TEST_DIR := tests
+
+# Colors for output
+RED := \033[0;31m
+GREEN := \033[0;32m
+YELLOW := \033[1;33m
+NC := \033[0m # No Color
+
+help: ## Show this help message
+	@echo "$(GREEN)IOCParser Development Commands$(NC)"
+	@echo ""
+	@grep -E '^[a-zA-Z_-]+:.*?## .*$$' $(MAKEFILE_LIST) | sort | awk 'BEGIN {FS = ":.*?## "}; {printf "$(YELLOW)%-20s$(NC) %s\n", $$1, $$2}'
+
+install: ## Install the package in production mode
+	$(PIP) install -e .
+	@echo "$(GREEN)✓ IOCParser installed successfully$(NC)"
+
+install-dev: ## Install the package with development dependencies
+	$(PIP) install -e ".[dev]"
+	pre-commit install
+	@echo "$(GREEN)✓ Development environment setup complete$(NC)"
+
+clean: ## Clean build artifacts and cache files
+	rm -rf build/ dist/ *.egg-info
+	rm -rf .coverage htmlcov/ .pytest_cache/
+	rm -rf .mypy_cache/ .ruff_cache/
+	find . -type d -name __pycache__ -exec rm -rf {} + 2>/dev/null || true
+	find . -type f -name "*.pyc" -delete
+	find . -type f -name "*.pyo" -delete
+	find . -type f -name "*~" -delete
+	@echo "$(GREEN)✓ Cleaned build artifacts and cache files$(NC)"
+
+test: ## Run all tests with coverage
+	pytest $(TEST_DIR) -v --cov=$(SRC_DIR) --cov-report=term-missing --cov-report=html
+	@echo "$(GREEN)✓ Tests completed. Coverage report available in htmlcov/index.html$(NC)"
+
+test-fast: ## Run tests without coverage (faster)
+	pytest $(TEST_DIR) -v
+	@echo "$(GREEN)✓ Tests completed$(NC)"
+
+test-benchmark: ## Run performance benchmark tests
+	pytest $(TEST_DIR)/test_performance.py -v --benchmark-only
+	@echo "$(GREEN)✓ Benchmark tests completed$(NC)"
+
+lint: ## Run all linters (ruff, mypy, pylint)
+	@echo "$(YELLOW)Running Ruff...$(NC)"
+	ruff check $(SRC_DIR) $(TEST_DIR)
+	@echo "$(YELLOW)Running MyPy...$(NC)"
+	mypy $(SRC_DIR)
+	@echo "$(YELLOW)Running Pylint...$(NC)"
+	pylint $(SRC_DIR) || true
+	@echo "$(GREEN)✓ Linting completed$(NC)"
+
+format: ## Auto-format code with ruff
+	ruff check --fix $(SRC_DIR) $(TEST_DIR)
+	ruff format $(SRC_DIR) $(TEST_DIR)
+	@echo "$(GREEN)✓ Code formatted$(NC)"
+
+security: ## Run security checks (bandit, safety)
+	@echo "$(YELLOW)Running Bandit security scan...$(NC)"
+	bandit -r $(SRC_DIR) -ll
+	@echo "$(YELLOW)Checking for known vulnerabilities in dependencies...$(NC)"
+	safety check --json || true
+	pip-audit || true
+	@echo "$(GREEN)✓ Security checks completed$(NC)"
+
+check-all: lint test security ## Run all checks (lint, test, security)
+	@echo "$(GREEN)✓ All checks completed successfully$(NC)"
+
+pre-commit: ## Run pre-commit hooks on all files
+	pre-commit run --all-files
+	@echo "$(GREEN)✓ Pre-commit hooks completed$(NC)"
+
+pre-commit-update: ## Update pre-commit hooks to latest versions
+	pre-commit autoupdate
+	@echo "$(GREEN)✓ Pre-commit hooks updated$(NC)"
+
+build: clean ## Build distribution packages
+	$(PYTHON) -m build
+	@echo "$(GREEN)✓ Distribution packages built in dist/$(NC)"
+
+publish-test: build ## Publish to TestPyPI (for testing)
+	$(PYTHON) -m twine upload --repository testpypi dist/*
+	@echo "$(GREEN)✓ Published to TestPyPI$(NC)"
+
+publish: build ## Publish to PyPI (requires credentials)
+	$(PYTHON) -m twine upload dist/*
+	@echo "$(GREEN)✓ Published to PyPI$(NC)"
+
+docs: ## Generate API documentation
+	pdoc --html --output-dir docs $(SRC_DIR) --force
+	@echo "$(GREEN)✓ Documentation generated in docs/$(NC)"
+
+quality-report: ## Generate comprehensive quality report
+	@echo "$(YELLOW)Generating Quality Report...$(NC)"
+	@echo "================================"
+	@echo "Code Coverage:"
+	@pytest $(TEST_DIR) --cov=$(SRC_DIR) --cov-report=term | grep TOTAL || true
+	@echo ""
+	@echo "Code Complexity:"
+	@ruff check $(SRC_DIR) --select C901 --statistics || true
+	@echo ""
+	@echo "Type Coverage:"
+	@mypy $(SRC_DIR) --html-report mypy_report 2>&1 | grep "Success:" || true
+	@echo ""
+	@echo "Security Issues:"
+	@bandit -r $(SRC_DIR) -f json 2>/dev/null | python -m json.tool | grep '"issue_severity"' | wc -l | xargs echo "Total issues found:" || true
+	@echo "$(GREEN)✓ Quality report completed$(NC)"
+
+init-misp: ## Initialize MISP warning lists
+	iocparser --init
+	@echo "$(GREEN)✓ MISP warning lists initialized$(NC)"
+
+example: ## Run IOCParser on the example file
+	iocparser -f examples/sample_report.txt --json
+	@echo "$(GREEN)✓ Example completed$(NC)"
+
+dev-install: install-dev ## Alias for install-dev
+
+setup: install-dev init-misp ## Complete setup (install + MISP lists)
+	@echo "$(GREEN)✓ Complete setup finished$(NC)"
+
+watch: ## Watch for file changes and run tests
+	watchmedo shell-command --patterns="*.py" --recursive --command="make test-fast" $(SRC_DIR) $(TEST_DIR)
+
+ci: ## Run CI pipeline locally
+	@echo "$(YELLOW)Running CI pipeline...$(NC)"
+	make lint
+	make test
+	make security
+	@echo "$(GREEN)✓ CI pipeline completed successfully$(NC)"
+
+stats: ## Show code statistics
+	@echo "$(YELLOW)Code Statistics:$(NC)"
+	@echo "Lines of code:"
+	@find $(SRC_DIR) -name "*.py" -exec wc -l {} + | tail -1
+	@echo "Number of Python files:"
+	@find $(SRC_DIR) -name "*.py" | wc -l
+	@echo "Number of tests:"
+	@grep -r "def test_" $(TEST_DIR) | wc -l
+	@echo "$(GREEN)✓ Statistics generated$(NC)"
+
+update-deps: ## Update all dependencies to latest versions
+	$(PIP) list --outdated
+	$(PIP) install --upgrade pip setuptools wheel
+	$(PIP) install --upgrade -r requirements.txt
+	@echo "$(GREEN)✓ Dependencies updated$(NC)"
+
+freeze: ## Freeze current dependencies
+	$(PIP) freeze > requirements-freeze.txt
+	@echo "$(GREEN)✓ Dependencies frozen to requirements-freeze.txt$(NC)"
+
+validate: ## Validate project configuration files
+	@echo "$(YELLOW)Validating configuration files...$(NC)"
+	python -m py_compile $(SRC_DIR)/*.py $(SRC_DIR)/**/*.py
+	python -c "import toml; toml.load('pyproject.toml')"
+	yamllint .pre-commit-config.yaml
+	yamllint .github/workflows/*.yml
+	@echo "$(GREEN)✓ All configuration files are valid$(NC)"
+
+# Default target
+.DEFAULT_GOAL := help

{iocparser_tool-2.0.0/iocparser_tool.egg-info → iocparser_tool-3.0.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: iocparser-tool
-Version: 2.0.0
+Version: 3.0.0
 Summary: A tool for extracting Indicators of Compromise from security reports
 Home-page: https://github.com/seifreed/iocparser
 Author: Marc Rivero
@@ -60,7 +60,7 @@ Dynamic: requires-python
 A tool for extracting Indicators of Compromise (IOCs) from security reports in HTML, PDF, and plain text formats.
 
 Author: Marc Rivero | @seifreed  
-Version: 1.0.1
+Version: 2.0.0
 
 ## Features