@jambudipa/spider 0.1.0 → 0.2.0

package/README.md

@@ -1,14 +1,40 @@
-# @jambudipa.io/spider
+# @jambudipa/spider
 
 A powerful, Effect.js-based web crawling framework for modern TypeScript applications. Built for type safety, composability, and enterprise-scale crawling operations.
 
+## 🏆 **Battle-Tested Against Real-World Scenarios**
+
+**Spider successfully handles ALL 16 https://web-scraping.dev challenge scenarios** - the most comprehensive web scraping test suite available:
+
+| ✅ Scenario | Description | Complexity |
+|-------------|-------------|------------|
+| **Static Paging** | Traditional pagination navigation | Basic |
+| **Endless Scroll** | Infinite scroll content loading | Dynamic |
+| **Button Loading** | Dynamic content via button clicks | Dynamic |
+| **GraphQL Requests** | Background API data fetching | Advanced |
+| **Hidden Data** | Extracting non-visible content | Intermediate |
+| **Product Markup** | Structured data extraction | Intermediate |
+| **Local Storage** | Browser storage interaction | Advanced |
+| **Secret API Tokens** | Authentication handling | Security |
+| **CSRF Protection** | Token-based security bypass | Security |
+| **Cookie Authentication** | Session-based access control | Security |
+| **PDF Downloads** | Binary file handling | Special |
+| **Cookie Popups** | Modal interaction handling | Special |
+| **New Tab Links** | Multi-tab navigation | Special |
+| **Block Pages** | Anti-bot detection handling | Anti-Block |
+| **Invalid Referer Blocking** | Header-based access control | Anti-Block |
+| **Persistent Cookie Blocking** | Long-term blocking mechanisms | Anti-Block |
+
+🎯 **[View Live Test Results](https://github.com/jambudipa/spider/actions)** | 📊 **100% Test Pass Rate** | 🚀 **Production Ready**
+
 ## ✨ Key Features
 
 - **🔥 Effect.js Foundation**: Type-safe, functional composition with robust error handling
 - **⚡ High Performance**: Concurrent crawling with intelligent worker pool management  
 - **🤖 Robots.txt Compliant**: Automatic robots.txt parsing and compliance checking
 - **🔄 Resumable Crawls**: State persistence and crash recovery capabilities
-- **🛡️ Middleware System**: Extensible middleware for rate limiting, authentication, and custom processing
+- **🛡️ Anti-Bot Bypass**: Handles complex blocking mechanisms and security measures
+- **🌐 Browser Automation**: Playwright integration for JavaScript-heavy sites
 - **📊 Built-in Monitoring**: Comprehensive logging and performance monitoring
 - **🎯 TypeScript First**: Full type safety with excellent IntelliSense support
 
@@ -49,18 +75,18 @@ Effect.runPromise(program.pipe(
 
 ### 🆕 New to Spider?
 - **[Getting Started Guide](./docs/guides/getting-started.md)** - Complete setup and first crawl
-- **[Configuration Guide](./docs/guides/configuration.md)** - Customise Spider for your needs
-- **[Basic Examples](./docs/examples/basic-crawling.md)** - Working examples to get you started
+- **[Examples](./docs/examples/)** - Working examples to get you started
+- **[Basic Configuration](./docs/guides/configuration.md)** - Configuration options
 
-### 🔄 Migrating from Another Library?
-- **[Migration Guide](./docs/guides/migration.md)** - Move from Puppeteer, Playwright, or Scrapy
-- **[Advanced Patterns](./docs/guides/advanced-patterns.md)** - Implement sophisticated crawling logic
-- **[Performance Guide](./docs/guides/performance.md)** - Optimise for your use case
+### 🔄 Advanced Usage
+- **[Browser Automation](./docs/guides/browser-automation.md)** - Handle dynamic content
+- **[Anti-Bot Protection](./docs/guides/anti-bot.md)** - Bypass blocking mechanisms
+- **[Security Handling](./docs/guides/security.md)** - Authentication and sessions
 
 ### 🏭 Building Production Systems?
-- **[Enterprise Patterns](./docs/examples/enterprise-patterns.md)** - Production-ready crawling solutions
-- **[Monitoring Guide](./docs/features/monitoring.md)** - Set up observability and alerting
+- **[Performance Guide](./docs/guides/performance.md)** - Scale your crawling operations
 - **[API Reference](./docs/api/)** - Complete technical documentation
+- **[Enterprise Patterns](./docs/examples/enterprise-patterns.md)** - Production-ready patterns
 
 ## 🛠️ Quick Configuration
 
@@ -83,7 +109,7 @@ const config = makeSpiderConfig({
 The spider can be configured for different scraping scenarios:
 
 ```typescript
-import { makeSpiderConfig } from '@jambudipa.io/spider';
+import { makeSpiderConfig } from '@jambudipa/spider';
 
 const config = makeSpiderConfig({
   // Basic settings
@@ -118,7 +144,7 @@ import {
   LoggingMiddleware,
   RateLimitMiddleware,
   UserAgentMiddleware 
-} from '@jambudipa.io/spider';
+} from '@jambudipa/spider';
 
 const middlewares = new MiddlewareManager()
   .use(new LoggingMiddleware({ level: 'info' }))
@@ -142,7 +168,7 @@ import {
   SpiderService, 
   ResumabilityService,
   FileStorageBackend 
-} from '@jambudipa.io/spider';
+} from '@jambudipa/spider';
 import { Effect, Layer } from 'effect';
 
 // Configure resumability with file storage
@@ -191,7 +217,7 @@ const program = Effect.gen(function* () {
 Extract and process links from pages:
 
 ```typescript
-import { LinkExtractorService } from '@jambudipa.io/spider';
+import { LinkExtractorService } from '@jambudipa/spider';
 
 const program = Effect.gen(function* () {
   const linkExtractor = yield* LinkExtractorService;
@@ -260,7 +286,7 @@ const program = Effect.gen(function* () {
 The library uses Effect for comprehensive error handling:
 
 ```typescript
-import { NetworkError, ResponseError, RobotsTxtError } from '@jambudipa.io/spider';
+import { NetworkError, ResponseError, RobotsTxtError } from '@jambudipa/spider';
 
 const program = Effect.gen(function* () {
   const spider = yield* SpiderService;
@@ -295,7 +321,7 @@ const program = Effect.gen(function* () {
 Create custom middleware for specific needs:
 
 ```typescript
-import { SpiderMiddleware, SpiderRequest, SpiderResponse } from '@jambudipa.io/spider';
+import { SpiderMiddleware, SpiderRequest, SpiderResponse } from '@jambudipa/spider';
 import { Effect } from 'effect';
 
 class CustomAuthMiddleware implements SpiderMiddleware {
@@ -326,7 +352,7 @@ const middlewares = new MiddlewareManager()
 Monitor scraping performance:
 
 ```typescript
-import { WorkerHealthMonitorService } from '@jambudipa.io/spider';
+import { WorkerHealthMonitorService } from '@jambudipa/spider';
 
 const program = Effect.gen(function* () {
   const healthMonitor = yield* WorkerHealthMonitorService;
@@ -347,18 +373,6 @@ const program = Effect.gen(function* () {
 });
 ```
 
-## Contributing
-
-1. Fork the repository
-2. Create a feature branch: `git checkout -b feature/new-feature`
-3. Make your changes
-4. Add tests for new functionality
-5. Run tests: `npm test`
-6. Run linting: `npm run lint`
-7. Commit changes: `git commit -am 'Add new feature'`
-8. Push to branch: `git push origin feature/new-feature`
-9. Submit a pull request
-
 ## Development
 
 ```bash
@@ -388,17 +402,6 @@ npm run format
 
 MIT License - see [LICENSE](LICENSE) file for details.
 
-## Changelog
-
-### 1.0.0
-- Initial standalone release
-- Migrated from monorepo structure
-- Full TypeScript support
-- Comprehensive middleware system
-- Resumable scraping functionality
-- Multiple storage backends
-- Rate limiting and performance monitoring
-
 ## 📚 Documentation
 
 Comprehensive documentation is available in the [`/docs`](./docs) directory:
@@ -406,7 +409,6 @@ Comprehensive documentation is available in the [`/docs`](./docs) directory:
 ### 🚀 Quick Links
 - **[Getting Started Guide](./docs/guides/getting-started.md)** - Installation, setup, and first crawl
 - **[API Reference](./docs/api/)** - Complete API documentation
-- **[Configuration Guide](./docs/guides/configuration.md)** - Configuration options and patterns
 - **[Examples](./docs/examples/)** - Working examples for common use cases
 
 ### 📖 Complete Documentation
@@ -417,10 +419,10 @@ Comprehensive documentation is available in the [`/docs`](./docs) directory:
 
 ## Support
 
-- [GitHub Issues](https://github.com/jambudipa-io/spider/issues)
+- [GitHub Issues](https://github.com/jambudipa/spider/issues)
 - [Complete Documentation](./docs/)
 - [Working Examples](./docs/examples/)
 
 ---
 
-Built with ❤️ by [Jambudipa.io](https://jambudipa.io)
+Built with ❤️ by [JAMBUDIPA](https://jambudipa.io)

package/dist/index.js

@@ -5,7 +5,7 @@ import * as path from "path";
 import * as fs$1 from "fs/promises";
 import { CookieJar } from "tough-cookie";
 class SpiderConfig extends Effect.Service()(
-  "@jambudipa.io/SpiderConfig",
+  "@jambudipa/spiderConfig",
   {
     effect: Effect.sync(() => makeSpiderConfig({}))
   }
@@ -1189,7 +1189,7 @@ class SpiderState extends Schema.Class("SpiderState")({
 }) {
 }
 class SpiderSchedulerService extends Effect.Service()(
-  "@jambudipa.io/SpiderSchedulerService",
+  "@jambudipa/spiderSchedulerService",
   {
     effect: Effect.gen(function* () {
       const config = yield* SpiderConfig;
@@ -1365,7 +1365,7 @@ const SpiderScheduler_service = /* @__PURE__ */ Object.freeze(/* @__PURE__ */ Ob
   SpiderStateKey
 }, Symbol.toStringTag, { value: "Module" }));
 class SpiderService extends Effect.Service()(
-  "@jambudipa.io/Spider",
+  "@jambudipa/spider",
   {
     effect: Effect.gen(function* () {
       const robots = yield* RobotsService;