pagerts 0.2.0 → 0.4.1

package/README.md

@@ -1,39 +1,243 @@
-# README
+# PagerTS
 
-PagerTS is a command line utility that provides the user with a portable tool for transforming an URL into a JSON Object.
+[![CI/CD](https://github.com/akinevz0/pagerts/workflows/CI%2FCD%20Security%20Pipeline/badge.svg)](https://github.com/akinevz0/pagerts/actions)
+[![Security](https://img.shields.io/badge/security-maintained-green.svg)](./SECURITY.md)
+[![Node.js Version](https://img.shields.io/badge/node-%3E%3D18.0.0-brightgreen.svg)](https://nodejs.org)
+[![License](https://img.shields.io/badge/license-MIT-blue.svg)](./LICENSE)
 
-The output of this command contains all of navigable items one can navigate to within a webpage.
+PagerTS is a secure, modern command-line utility that transforms URLs into structured JSON objects, extracting all navigable items and resources from webpages.
+
+## Features
+
+- 🔒 **Security-First**: Built-in URL validation, rate limiting, and XSS protection
+- 🚀 **Modern TypeScript**: Strict type checking and modern ES2022 syntax
+- ⚡ **Fast**: Efficient parsing with JSDOM and concurrent request handling
+- 🧪 **Well-Tested**: Comprehensive test coverage with Jest
+- 📦 **Easy to Use**: Simple CLI interface with sensible defaults
+
+## Installation
+
+### Global Installation
+
+```bash
+npm install -g pagerts
+pagerts <url>
+```
+
+### Using npx (No Installation Required)
+
+```bash
+npx pagerts <url>
+```
+
+### From Source
+
+```bash
+git clone https://github.com/akinevz0/pagerts.git
+cd pagerts
+npm install
+npm run build
+npm link
+```
 
 ## Usage
 
-To use `pagerts` invoke it in the command line as:
+### Basic Usage
+
+Extract resources from a remote URL:
+
+```bash
+pagerts https://example.com
+```
+
+Extract from multiple URLs:
+
+```bash
+pagerts https://example.com https://example.org
+```
+
+Extract from a local HTML file:
 
 ```bash
-pagerts https://website/page.html
+pagerts file:///path/to/file.html
+```
+
+### Output Format
+
+The output is a JSON object containing:
+
+```json
+{
+  "title": "Page Title",
+  "url": "https://example.com",
+  "resources": [
+    {
+      "name": "Link Text",
+      "url": "https://example.com/page"
+    }
+  ]
+}
 ```
 
-There is also support for loading local system html resources.
+Fields:
+
+- `title`: The page's title extracted from the `<title>` tag
+- `url`: The URL of the page
+- `resources`: Array of resources found on the page (links, meta tags, embeds)
+  - `name`: Readable text or description
+  - `url`: Target URL of the resource
+
+## Security
+
+PagerTS takes security seriously. See [SECURITY.md](./SECURITY.md) for:
+
+- Security features and protections
+- How to report vulnerabilities
+- Best practices for users
+- Security checklist for contributors
 
-## Output
+### Built-in Security Features
 
-The output encodes a dependency list of resources. It is parsed using JSDOM library and returned to the user as an object annotated with fields `title`, `url`, `resources`.
+- ✅ URL validation (only allows `http://`, `https://`, `file://`)
+- ✅ Input sanitization to prevent XSS attacks
+- ✅ Rate limiting (50 requests/minute by default)
+- ✅ Request timeouts to prevent hanging
+- ✅ Maximum URL length enforcement
+- ✅ Suspicious pattern detection
+- ✅ Safe HTML parsing (no script execution)
 
-The last field represents a list of tuples, mapping name of the resource to a url.
+## Development
 
-The name is extracted from the readable text on the page.
+### Prerequisites
 
-## Installing
+- Node.js >= 18.0.0
+- npm >= 9.0.0
 
-The CLI can be installed using npm:
+### Setup
 
 ```bash
-npm i -g pagerts
-pagerts <url>
+# Clone the repository
+git clone https://github.com/akinevz0/pagerts.git
+cd pagerts
+
+# Install dependencies
+npm install
+
+# Run in development mode
+npm run dev <url>
 ```
 
-Or run from npx:
+### Available Scripts
 
 ```bash
-npx pagerts <url>
+# Run tests
+npm test
+
+# Run tests in watch mode
+npm test:watch
+
+# Build the project
+npm run build
+
+# Lint code
+npm run lint
+
+# Fix linting issues
+npm run lint:fix
+
+# Type check
+npm run type-check
+
+# Format code
+npm run format
+
+# Check formatting
+npm run format:check
+
+# Security audit
+npm run security:audit
+
+# Complete security check (audit + lint)
+npm run security:check
+```
+
+### Project Structure
+
 ```
+pagerts/
+├── src/
+│   ├── main.ts                 # CLI entry point
+│   ├── security.ts             # Security utilities
+│   ├── resource.ts             # Resource types
+│   ├── extractors/             # Content extractors
+│   │   ├── AbstractExtractor.ts
+│   │   ├── PageExtractor.ts
+│   │   ├── ResourceExtractor.ts
+│   │   └── TagExtractor.ts
+│   ├── page/                   # Page fetching
+│   │   ├── Page.ts
+│   │   └── PageFetcher.ts
+│   ├── printers/               # Output formatters
+│   │   ├── AbstractResourcePrinter.ts
+│   │   ├── JSONStylePrinter.ts
+│   │   └── LogStylePrinter.ts
+│   └── __tests__/              # Test files
+├── bin/                        # Built files
+├── .github/workflows/          # CI/CD pipelines
+├── package.json
+├── tsconfig.json
+├── jest.config.js
+├── eslint.config.js
+└── SECURITY.md
+```
+
+## Contributing
+
+Contributions are welcome! Please:
+
+1. Fork the repository
+2. Create a feature branch (`git checkout -b feature/amazing-feature`)
+3. Commit your changes (`git commit -m 'Add amazing feature'`)
+4. Push to the branch (`git push origin feature/amazing-feature`)
+5. Open a Pull Request
+
+### Contribution Guidelines
+
+- Write tests for new features
+- Follow the existing code style (enforced by ESLint and Prettier)
+- Update documentation as needed
+- Ensure all tests pass (`npm test`)
+- Run security checks (`npm run security:check`)
+- Follow security best practices (see [SECURITY.md](./SECURITY.md))
+
+## License
+
+This project is licensed under the MIT License - see the [LICENSE](./LICENSE) file for details.
+
+## Author
+
+**Kirill kn253 Nevzorov**
+
+## Support
+
+- 🐛 [Report bugs](https://github.com/akinevz0/pagerts/issues)
+- 💡 [Request features](https://github.com/akinevz0/pagerts/issues)
+- 🔒 [Report security issues](./SECURITY.md)
+
+## Changelog
+
+### v0.3.0 (Latest)
+
+- ✨ Added comprehensive security features
+- ✨ Implemented URL validation and sanitization
+- ✨ Added rate limiting
+- ✨ Modernized codebase with TypeScript strict mode
+- ✨ Added ESLint with security plugin
+- ✨ Added comprehensive test suite
+- ✨ Added CI/CD with GitHub Actions
+- ✨ Improved error handling and retry logic
+- 📚 Added security documentation
+
+### v0.2.0
 
+- Initial public release

package/SECURITY.md

@@ -0,0 +1,160 @@
+# Security Policy
+
+## Supported Versions
+
+We release patches for security vulnerabilities. Currently supported versions:
+
+| Version | Supported          |
+| ------- | ------------------ |
+| 0.3.x   | :white_check_mark: |
+| < 0.3.0 | :x:                |
+
+## Security Features
+
+PagerTS implements several security measures to protect users:
+
+### Input Validation
+
+- **URL Validation**: All URLs are validated before processing
+- **Protocol Restrictions**: Only `http://`, `https://`, and `file://` protocols are allowed
+- **Length Limits**: URLs are limited to 2048 characters to prevent DoS attacks
+- **Pattern Detection**: Suspicious patterns (javascript:, data:, etc.) are blocked
+
+### Rate Limiting
+
+- Requests are rate-limited to prevent abuse (default: 50 requests per minute)
+- Configurable rate limits per instance
+
+### Safe HTML Parsing
+
+- JSDOM is configured to run in secure mode
+- JavaScript execution from fetched pages is disabled
+- Timeouts prevent hanging on slow resources
+- Retry logic with exponential backoff for transient failures
+
+### Data Sanitization
+
+- HTML content is sanitized to prevent XSS attacks
+- Special characters are properly escaped in output
+
+## Reporting a Vulnerability
+
+We take the security of PagerTS seriously. If you believe you have found a security vulnerability, please report it to us as described below.
+
+**Please do not report security vulnerabilities through public GitHub issues.**
+
+Instead, please report them via email to the maintainer or through GitHub's private vulnerability reporting feature.
+
+Please include the following information:
+
+- Type of issue (e.g., buffer overflow, SQL injection, cross-site scripting, etc.)
+- Full paths of source file(s) related to the manifestation of the issue
+- The location of the affected source code (tag/branch/commit or direct URL)
+- Any special configuration required to reproduce the issue
+- Step-by-step instructions to reproduce the issue
+- Proof-of-concept or exploit code (if possible)
+- Impact of the issue, including how an attacker might exploit it
+
+### What to Expect
+
+- **Acknowledgment**: We will acknowledge your report within 48 hours
+- **Communication**: We will keep you informed about the progress of fixing the issue
+- **Credit**: We will give you credit for the discovery when we announce the fix (unless you prefer to remain anonymous)
+
+## Security Best Practices for Users
+
+When using PagerTS, follow these security guidelines:
+
+### 1. Be Cautious with URLs
+
+```bash
+# Good - trusted domain
+pagerts https://example.com
+
+# Bad - suspicious or untrusted URLs
+pagerts javascript:alert(1)  # Will be blocked
+pagerts data:text/html,...   # Will be blocked
+```
+
+### 2. Use Environment Variables for Sensitive Data
+
+Never hardcode sensitive information. Use environment variables:
+
+```bash
+# Create a .env file (never commit this!)
+API_KEY=your_secret_key
+
+# Use it in your scripts
+pagerts $TARGET_URL
+```
+
+### 3. Validate Output
+
+Always validate and sanitize output before using it in other systems:
+
+```bash
+# Pipe through jq for safe JSON processing
+pagerts https://example.com | jq '.'
+```
+
+### 4. Keep Dependencies Updated
+
+Regularly update PagerTS and its dependencies:
+
+```bash
+npm update -g pagerts
+```
+
+### 5. Network Security
+
+- Use HTTPS URLs whenever possible
+- Be cautious when fetching from local networks
+- Consider using a VPN or proxy for sensitive operations
+
+### 6. File System Access
+
+When using `file://` URLs:
+
+- Ensure you have appropriate permissions
+- Be cautious with symbolic links
+- Validate file paths to prevent directory traversal
+
+## Security Checklist for Contributors
+
+If you're contributing to PagerTS, ensure your code:
+
+- [ ] Validates all user input
+- [ ] Uses parameterized queries (if applicable)
+- [ ] Properly escapes output
+- [ ] Handles errors gracefully without exposing sensitive information
+- [ ] Includes tests for security-critical functionality
+- [ ] Doesn't introduce new dependencies without security review
+- [ ] Follows the principle of least privilege
+- [ ] Includes appropriate logging (without logging sensitive data)
+
+## Dependencies
+
+PagerTS regularly audits its dependencies for security vulnerabilities. Run the security check:
+
+```bash
+npm run security:check
+```
+
+## Automated Security Testing
+
+PagerTS uses:
+
+- **npm audit**: Checks for known vulnerabilities in dependencies
+- **ESLint with security plugin**: Static analysis for security issues
+- **GitHub Dependabot**: Automated dependency updates
+- **GitHub Actions**: CI/CD with security scanning
+
+## Contact
+
+For security concerns, contact: [GitHub Issues](https://github.com/akinevz0/pagerts/issues)
+
+## Acknowledgments
+
+We thank the following researchers for responsibly disclosing vulnerabilities:
+
+- (None yet - be the first!)