@jambudipa/spider 0.1.0 → 0.2.1

package/README.md

@@ -1,14 +1,57 @@
-# @jambudipa.io/spider
-
-A powerful, Effect.js-based web crawling framework for modern TypeScript applications. Built for type safety, composability, and enterprise-scale crawling operations.
+# @jambudipa/spider
+
+[![CI Status](https://github.com/jambudipa/spider/workflows/Spider%20Scenario%20Tests/badge.svg)](https://github.com/jambudipa/spider/actions)
+[![Coverage](https://codecov.io/gh/jambudipa/spider/branch/main/graph/badge.svg)](https://codecov.io/gh/jambudipa/spider)
+[![npm version](https://badge.fury.io/js/@jambudipa%2Fspider.svg)](https://badge.fury.io/js/@jambudipa%2Fspider)
+[![Node.js Version](https://img.shields.io/node/v/@jambudipa/spider.svg)](https://nodejs.org/)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+
+A powerful, Effect-based web crawling framework for modern TypeScript applications. Built for type safety, composability, and enterprise-scale crawling operations.
+
+> **⚠️ Pre-Release API**: Spider is currently in pre-release development (v0.x.x). The API may change frequently as we refine the library towards a stable v1.0.0 release. Consider this when using Spider in production environments and expect potential breaking changes in minor version updates.
+
+## 🏆 **Battle-Tested Against Real-World Scenarios**
+
+**Spider successfully handles ALL 16 https://web-scraping.dev challenge scenarios** - the most comprehensive web scraping test suite available:
+
+| ✅ Scenario | Description | Complexity |
+|-------------|-------------|------------|
+| **Static Paging** | Traditional pagination navigation | Basic |
+| **Endless Scroll** | Infinite scroll content loading | Dynamic |
+| **Button Loading** | Dynamic content via button clicks | Dynamic |
+| **GraphQL Requests** | Background API data fetching | Advanced |
+| **Hidden Data** | Extracting non-visible content | Intermediate |
+| **Product Markup** | Structured data extraction | Intermediate |
+| **Local Storage** | Browser storage interaction | Advanced |
+| **Secret API Tokens** | Authentication handling | Security |
+| **CSRF Protection** | Token-based security bypass | Security |
+| **Cookie Authentication** | Session-based access control | Security |
+| **PDF Downloads** | Binary file handling | Special |
+| **Cookie Popups** | Modal interaction handling | Special |
+| **New Tab Links** | Multi-tab navigation | Special |
+| **Block Pages** | Anti-bot detection handling | Anti-Block |
+| **Invalid Referer Blocking** | Header-based access control | Anti-Block |
+| **Persistent Cookie Blocking** | Long-term blocking mechanisms | Anti-Block |
+
+🎯 **[View Live Test Results](https://github.com/jambudipa/spider/actions/workflows/ci.yml)** | 📊 **All Scenario Tests Passing** | 🚀 **Production Ready**
+
+> **Live Testing**: Our CI pipeline runs all 16 web scraping scenarios against real websites daily, ensuring Spider remains robust against changing web technologies.
+
+### 🔍 **Current Status** (Updated: Aug 2025)
+- ✅ **Core Functionality**: All web scraping scenarios working
+- ✅ **Type Safety**: Full TypeScript compilation without errors
+- ✅ **Build System**: Package builds successfully for distribution
+- ✅ **Test Suite**: 92+ scenario tests passing against live websites
+- ⚠️ **Code Quality**: 1,163 linting issues identified (technical debt - does not affect functionality)
 
 ## ✨ Key Features
 
-- **🔥 Effect.js Foundation**: Type-safe, functional composition with robust error handling
+- **🔥 Effect Foundation**: Type-safe, functional composition with robust error handling
 - **⚡ High Performance**: Concurrent crawling with intelligent worker pool management  
 - **🤖 Robots.txt Compliant**: Automatic robots.txt parsing and compliance checking
 - **🔄 Resumable Crawls**: State persistence and crash recovery capabilities
-- **🛡️ Middleware System**: Extensible middleware for rate limiting, authentication, and custom processing
+- **🛡️ Anti-Bot Bypass**: Handles complex blocking mechanisms and security measures
+- **🌐 Browser Automation**: Playwright integration for JavaScript-heavy sites
 - **📊 Built-in Monitoring**: Comprehensive logging and performance monitoring
 - **🎯 TypeScript First**: Full type safety with excellent IntelliSense support
 
@@ -45,22 +88,30 @@ Effect.runPromise(program.pipe(
 ))
 ```
 
-## 🎯 What's Next?
+## 📚 Documentation
+
+**Comprehensive documentation is now available** following the [Diátaxis framework](https://diataxis.fr/) for better learning and reference:
+
+### 🎓 New to Spider?
+Start with our **[Tutorial](./docs/tutorial/getting-started.md)** - a hands-on guide that takes you from installation to building advanced scrapers.
+
+### 📋 Need to solve a specific problem?
+Check our **[How-to Guides](./docs/how-to/)** for targeted solutions:
+- **[Authentication](./docs/how-to/authentication.md)** - Handle logins, sessions, and auth flows
+- **[Data Extraction](./docs/how-to/data-extraction.md)** - Extract structured data from HTML
+- **[Resumable Operations](./docs/how-to/resumable-operations.md)** - Build fault-tolerant crawlers
 
-### 🆕 New to Spider?
-- **[Getting Started Guide](./docs/guides/getting-started.md)** - Complete setup and first crawl
-- **[Configuration Guide](./docs/guides/configuration.md)** - Customise Spider for your needs
-- **[Basic Examples](./docs/examples/basic-crawling.md)** - Working examples to get you started
+### 📚 Need technical details?
+See our **[Reference Documentation](./docs/reference/)**:
+- **[API Reference](./docs/reference/api-reference.md)** - Complete API documentation
+- **[Configuration](./docs/reference/configuration.md)** - All configuration options
 
-### 🔄 Migrating from Another Library?
-- **[Migration Guide](./docs/guides/migration.md)** - Move from Puppeteer, Playwright, or Scrapy
-- **[Advanced Patterns](./docs/guides/advanced-patterns.md)** - Implement sophisticated crawling logic
-- **[Performance Guide](./docs/guides/performance.md)** - Optimise for your use case
+### 🧠 Want to understand the design?
+Read our **[Explanations](./docs/explanation/)**:
+- **[Architecture](./docs/explanation/architecture.md)** - System design and philosophy
+- **[Web Scraping Concepts](./docs/explanation/web-scraping-concepts.md)** - Core principles
 
-### 🏭 Building Production Systems?
-- **[Enterprise Patterns](./docs/examples/enterprise-patterns.md)** - Production-ready crawling solutions
-- **[Monitoring Guide](./docs/features/monitoring.md)** - Set up observability and alerting
-- **[API Reference](./docs/api/)** - Complete technical documentation
+**📖 [Browse All Documentation →](./docs/README.md)**
 
 ## 🛠️ Quick Configuration
 
@@ -83,7 +134,7 @@ const config = makeSpiderConfig({
 The spider can be configured for different scraping scenarios:
 
 ```typescript
-import { makeSpiderConfig } from '@jambudipa.io/spider';
+import { makeSpiderConfig } from '@jambudipa/spider';
 
 const config = makeSpiderConfig({
   // Basic settings
@@ -118,7 +169,7 @@ import {
   LoggingMiddleware,
   RateLimitMiddleware,
   UserAgentMiddleware 
-} from '@jambudipa.io/spider';
+} from '@jambudipa/spider';
 
 const middlewares = new MiddlewareManager()
   .use(new LoggingMiddleware({ level: 'info' }))
@@ -142,7 +193,7 @@ import {
   SpiderService, 
   ResumabilityService,
   FileStorageBackend 
-} from '@jambudipa.io/spider';
+} from '@jambudipa/spider';
 import { Effect, Layer } from 'effect';
 
 // Configure resumability with file storage
@@ -191,7 +242,7 @@ const program = Effect.gen(function* () {
 Extract and process links from pages:
 
 ```typescript
-import { LinkExtractorService } from '@jambudipa.io/spider';
+import { LinkExtractorService } from '@jambudipa/spider';
 
 const program = Effect.gen(function* () {
   const linkExtractor = yield* LinkExtractorService;
@@ -260,7 +311,7 @@ const program = Effect.gen(function* () {
 The library uses Effect for comprehensive error handling:
 
 ```typescript
-import { NetworkError, ResponseError, RobotsTxtError } from '@jambudipa.io/spider';
+import { NetworkError, ResponseError, RobotsTxtError } from '@jambudipa/spider';
 
 const program = Effect.gen(function* () {
   const spider = yield* SpiderService;
@@ -295,7 +346,7 @@ const program = Effect.gen(function* () {
 Create custom middleware for specific needs:
 
 ```typescript
-import { SpiderMiddleware, SpiderRequest, SpiderResponse } from '@jambudipa.io/spider';
+import { SpiderMiddleware, SpiderRequest, SpiderResponse } from '@jambudipa/spider';
 import { Effect } from 'effect';
 
 class CustomAuthMiddleware implements SpiderMiddleware {
@@ -326,7 +377,7 @@ const middlewares = new MiddlewareManager()
 Monitor scraping performance:
 
 ```typescript
-import { WorkerHealthMonitorService } from '@jambudipa.io/spider';
+import { WorkerHealthMonitorService } from '@jambudipa/spider';
 
 const program = Effect.gen(function* () {
   const healthMonitor = yield* WorkerHealthMonitorService;
@@ -347,18 +398,6 @@ const program = Effect.gen(function* () {
 });
 ```
 
-## Contributing
-
-1. Fork the repository
-2. Create a feature branch: `git checkout -b feature/new-feature`
-3. Make your changes
-4. Add tests for new functionality
-5. Run tests: `npm test`
-6. Run linting: `npm run lint`
-7. Commit changes: `git commit -am 'Add new feature'`
-8. Push to branch: `git push origin feature/new-feature`
-9. Submit a pull request
-
 ## Development
 
 ```bash
@@ -368,59 +407,68 @@ npm install
 # Build the package
 npm run build
 
-# Run tests
+# Run tests (all scenarios)
 npm test
 
 # Run tests with coverage
 npm run test:coverage
 
-# Type checking
+# Type checking (must pass)
 npm run typecheck
 
-# Linting
+# Validate CI setup locally
+npm run ci:validate
+
+# Code quality (has known issues)
+npm run lint        # Shows 1,163 issues
+npm run format     # Formats code consistently
+```
+
+### 🛠️ Contributing & Code Quality
+
+**Current State**: The codebase is fully functional with comprehensive test coverage, but has technical debt in code style consistency.
+
+- ✅ **Functional Changes**: All PRs must pass scenario tests
+- ✅ **Type Safety**: TypeScript compilation must succeed  
+- ✅ **Build System**: Package must build without errors
+- 🔄 **Code Style**: Help wanted fixing linting issues (great first contribution!)
+
+**Contributing to Code Quality**:
+```bash
+# See specific linting issues
 npm run lint
 
-# Format code
-npm run format
+# Fix auto-fixable issues
+npm run lint:fix
+
+# Focus areas for improvement:
+# - Unused variable cleanup (877 issues)
+# - Return type annotations (286 issues)  
+# - Nullish coalescing operators
+# - Console.log removal in production code
 ```
 
 ## License
 
 MIT License - see [LICENSE](LICENSE) file for details.
 
-## Changelog
-
-### 1.0.0
-- Initial standalone release
-- Migrated from monorepo structure
-- Full TypeScript support
-- Comprehensive middleware system
-- Resumable scraping functionality
-- Multiple storage backends
-- Rate limiting and performance monitoring
-
-## 📚 Documentation
+## 📚 Complete Documentation
 
-Comprehensive documentation is available in the [`/docs`](./docs) directory:
+All documentation is organized in the [`/docs`](./docs/) directory following the [Diátaxis framework](https://diataxis.fr/):
 
-### 🚀 Quick Links
-- **[Getting Started Guide](./docs/guides/getting-started.md)** - Installation, setup, and first crawl
-- **[API Reference](./docs/api/)** - Complete API documentation
-- **[Configuration Guide](./docs/guides/configuration.md)** - Configuration options and patterns
-- **[Examples](./docs/examples/)** - Working examples for common use cases
+- **🎓 [Tutorial](./docs/tutorial/)** - Learning-oriented lessons for getting started
+- **📋 [How-to Guides](./docs/how-to/)** - Problem-solving guides for specific tasks
+- **📚 [Reference](./docs/reference/)** - Technical reference and API documentation  
+- **🧠 [Explanation](./docs/explanation/)** - Understanding-oriented documentation
 
-### 📖 Complete Documentation
-- **[Documentation Index](./docs/README.md)** - Overview of all available documentation
-- **[User Guides](./docs/guides/)** - Step-by-step tutorials and best practices
-- **[Feature Documentation](./docs/features/)** - Deep dives into key capabilities
-- **[Advanced Examples](./docs/examples/)** - Real-world usage patterns
+**📖 [Start with the Documentation Index →](./docs/README.md)**
 
 ## Support
 
-- [GitHub Issues](https://github.com/jambudipa-io/spider/issues)
-- [Complete Documentation](./docs/)
-- [Working Examples](./docs/examples/)
+- [GitHub Issues](https://github.com/jambudipa/spider/issues) - Bug reports and feature requests
+- [Documentation](./docs/) - Comprehensive guides and reference material
+- [Tutorial](./docs/tutorial/getting-started.md) - Step-by-step learning guide
 
 ---
 
-Built with ❤️ by [Jambudipa.io](https://jambudipa.io)
+Built with ❤️ by [JAMBUDIPA](https://jambudipa.io)