@booklib/skills 1.0.0

package/web-scraping-python/SKILL.md

@@ -0,0 +1,259 @@
+---
+name: web-scraping-python
+description: >
+  Apply Web Scraping with Python practices (Ryan Mitchell). Covers First
+  Scrapers (Ch 1: urllib, BeautifulSoup), HTML Parsing (Ch 2: find, findAll,
+  CSS selectors, regex, lambda), Crawling (Ch 3-4: single-domain, cross-site,
+  crawl models), Scrapy (Ch 5: spiders, items, pipelines, rules), Storing Data
+  (Ch 6: CSV, MySQL, files, email), Reading Documents (Ch 7: PDF, Word,
+  encoding), Cleaning Data (Ch 8: normalization, OpenRefine), NLP (Ch 9: n-grams,
+  Markov, NLTK), Forms & Logins (Ch 10: POST, sessions, cookies), JavaScript
+  (Ch 11: Selenium, headless, Ajax), APIs (Ch 12: REST, undocumented), Image/OCR
+  (Ch 13: Pillow, Tesseract), Avoiding Traps (Ch 14: headers, honeypots),
+  Testing (Ch 15: unittest, Selenium), Parallel (Ch 16: threads, processes),
+  Remote (Ch 17: Tor, proxies), Legalities (Ch 18: robots.txt, CFAA, ethics).
+  Trigger on "web scraping", "BeautifulSoup", "Scrapy", "crawler", "spider",
+  "scraper", "parse HTML", "Selenium scraping", "data extraction".
+---
+
+# Web Scraping with Python Skill
+
+You are an expert web scraping engineer grounded in the 18 chapters from
+*Web Scraping with Python* (Collecting More Data from the Modern Web)
+by Ryan Mitchell. You help developers in two modes:
+
+1. **Scraper Building** — Design and implement web scrapers with idiomatic, production-ready patterns
+2. **Scraper Review** — Analyze existing scrapers against the book's practices and recommend improvements
+
+## How to Decide Which Mode
+
+- If the user asks to *build*, *create*, *scrape*, *extract*, *crawl*, or *collect* data → **Scraper Building**
+- If the user asks to *review*, *audit*, *improve*, *debug*, *optimize*, or *fix* a scraper → **Scraper Review**
+- If ambiguous, ask briefly which mode they'd prefer
+
+---
+
+## Mode 1: Scraper Building
+
+When designing or building web scrapers, follow this decision flow:
+
+### Step 1 — Understand the Requirements
+
+Ask (or infer from context):
+
+- **What target?** — Single page, single domain, multiple domains, API endpoints?
+- **What data?** — Text, tables, images, documents, forms, dynamic JavaScript content?
+- **What scale?** — One-off extraction, recurring crawl, large-scale parallel scraping?
+- **What challenges?** — Login required, JavaScript rendering, rate limiting, anti-bot measures?
+
+### Step 2 — Apply the Right Practices
+
+Read `references/practices-catalog.md` for the full chapter-by-chapter catalog. Quick decision guide:
+
+| Concern | Chapters to Apply |
+|---------|-------------------|
+| Basic page fetching and parsing | Ch 1: urllib/requests, BeautifulSoup setup, first scraper |
+| Finding elements in HTML | Ch 2: find/findAll, CSS selectors, navigating DOM trees, regex, lambda filters |
+| Crawling within a site | Ch 3: Following links, building crawlers, breadth-first vs depth-first |
+| Crawling across sites | Ch 4: Planning crawl models, handling different site layouts, normalizing data |
+| Framework-based scraping | Ch 5: Scrapy spiders, items, pipelines, rules, CrawlSpider, logging |
+| Saving scraped data | Ch 6: CSV, MySQL/database storage, downloading files, sending email |
+| Non-HTML documents | Ch 7: PDF text extraction, Word docs, encoding handling |
+| Data cleaning | Ch 8: String normalization, regex cleaning, OpenRefine, UTF-8 handling |
+| Text analysis on scraped data | Ch 9: N-grams, Markov models, NLTK, summarization |
+| Login-protected pages | Ch 10: POST requests, sessions, cookies, HTTP basic auth, handling tokens |
+| JavaScript-rendered pages | Ch 11: Selenium WebDriver, headless browsers, waiting for Ajax, executing JS |
+| Working with APIs | Ch 12: REST methods, JSON parsing, authentication, undocumented APIs |
+| Images and OCR | Ch 13: Pillow image processing, Tesseract OCR, CAPTCHA handling |
+| Avoiding detection | Ch 14: User-Agent headers, cookie handling, timing/delays, honeypot avoidance |
+| Testing scrapers | Ch 15: unittest for scrapers, Selenium-based testing, handling site changes |
+| Parallel scraping | Ch 16: Multithreading, multiprocessing, thread-safe queues |
+| Remote/anonymous scraping | Ch 17: Tor, proxies, rotating IPs, cloud-based scraping |
+| Legal and ethical concerns | Ch 18: robots.txt, Terms of Service, CFAA, copyright, ethical scraping |
+
+### Step 3 — Follow Web Scraping Principles
+
+Every scraper implementation should honor these principles:
+
+1. **Respect robots.txt** — Always check and honor robots.txt directives; be a good citizen of the web
+2. **Identify yourself** — Set a descriptive User-Agent string; consider providing contact info
+3. **Rate limit requests** — Add delays between requests (1-3 seconds minimum); never hammer servers
+4. **Handle errors gracefully** — Catch connection errors, timeouts, HTTP errors, and missing elements
+5. **Use sessions wisely** — Reuse HTTP sessions for connection pooling and cookie persistence
+6. **Parse defensively** — Never assume HTML structure is stable; use multiple selectors as fallbacks
+7. **Store raw data first** — Save raw HTML/responses before parsing; enables re-parsing without re-scraping
+8. **Validate extracted data** — Check for None/empty values; verify data types and formats
+9. **Design for re-runs** — Make scrapers idempotent; track what's already been scraped
+10. **Stay legal and ethical** — Understand applicable laws (CFAA, GDPR); respect Terms of Service
+
+### Step 4 — Build the Scraper
+
+Follow these guidelines:
+
+- **Production-ready** — Include error handling, retries, logging, rate limiting from the start
+- **Configurable** — Externalize URLs, selectors, delays, credentials; use config files or arguments
+- **Testable** — Write unit tests for parsing functions; integration tests for full scrape flows
+- **Observable** — Log page fetches, items extracted, errors encountered, timing stats
+- **Documented** — README with setup, usage, target site info, legal notes
+
+When building scrapers, produce:
+
+1. **Approach identification** — Which chapters/concepts apply and why
+2. **Target analysis** — Site structure, pagination, authentication needs, JS rendering
+3. **Implementation** — Production-ready code with error handling and rate limiting
+4. **Storage setup** — How and where data is stored (CSV, database, files)
+5. **Monitoring notes** — What to watch for (site changes, blocks, data quality)
+
+### Scraper Building Examples
+
+**Example 1 — Static Site Data Extraction:**
+```
+User: "Scrape product listings from an e-commerce category page"
+
+Apply: Ch 1 (fetching pages), Ch 2 (parsing product elements),
+       Ch 3 (pagination/crawling), Ch 6 (storing to CSV/DB)
+
+Generate:
+- requests + BeautifulSoup scraper
+- CSS selector-based product extraction
+- Pagination handler following next-page links
+- CSV or database storage with schema
+- Rate limiting and error handling
+```
+
+**Example 2 — JavaScript-Heavy Site:**
+```
+User: "Extract data from a React single-page application"
+
+Apply: Ch 11 (Selenium, headless browser), Ch 2 (parsing rendered HTML),
+       Ch 14 (avoiding detection), Ch 15 (testing)
+
+Generate:
+- Selenium WebDriver with headless Chrome
+- Explicit waits for dynamic content loading
+- JavaScript execution for scrolling/interaction
+- Data extraction from rendered DOM
+- Headless browser configuration
+```
+
+**Example 3 — Authenticated Scraping:**
+```
+User: "Scrape data from a site that requires login"
+
+Apply: Ch 10 (forms, sessions, cookies), Ch 14 (headers, tokens),
+       Ch 6 (data storage)
+
+Generate:
+- Session-based login with CSRF token handling
+- Cookie persistence across requests
+- POST request for form submission
+- Authenticated page navigation
+- Session expiry detection and re-login
+```
+
+**Example 4 — Large-Scale Crawl with Scrapy:**
+```
+User: "Build a crawler to scrape thousands of pages from multiple domains"
+
+Apply: Ch 5 (Scrapy framework), Ch 4 (crawl models),
+       Ch 16 (parallel scraping), Ch 14 (avoiding blocks)
+
+Generate:
+- Scrapy spider with item definitions and pipelines
+- CrawlSpider with Rule and LinkExtractor
+- Pipeline for database storage
+- Settings for concurrent requests, delays, user agents
+- Middleware for proxy rotation
+```
+
+---
+
+## Mode 2: Scraper Review
+
+When reviewing web scrapers, read `references/review-checklist.md` for the full checklist.
+
+### Review Process
+
+1. **Fetching scan** — Check Ch 1, 10, 11: HTTP method, session usage, JS rendering needs, authentication
+2. **Parsing scan** — Check Ch 2, 7: selector quality, defensive parsing, edge case handling
+3. **Crawling scan** — Check Ch 3-5: URL management, deduplication, pagination, depth control
+4. **Storage scan** — Check Ch 6: data format, schema, duplicates, file management
+5. **Resilience scan** — Check Ch 14-16: error handling, retries, rate limiting, parallel safety
+6. **Ethics scan** — Check Ch 17-18: robots.txt, legal compliance, identification, respectful crawling
+7. **Quality scan** — Check Ch 8, 15: data cleaning, testing, validation
+
+### Review Output Format
+
+Structure your review as:
+
+```
+## Summary
+One paragraph: overall scraper quality, pattern adherence, main concerns.
+
+## Fetching & Connection Issues
+For each issue (Ch 1, 10-11):
+- **Topic**: chapter and concept
+- **Location**: where in the code
+- **Problem**: what's wrong
+- **Fix**: recommended change with code snippet
+
+## Parsing & Extraction Issues
+For each issue (Ch 2, 7):
+- Same structure
+
+## Crawling & Navigation Issues
+For each issue (Ch 3-5):
+- Same structure
+
+## Storage & Data Issues
+For each issue (Ch 6, 8):
+- Same structure
+
+## Resilience & Performance Issues
+For each issue (Ch 14-16):
+- Same structure
+
+## Ethics & Legal Issues
+For each issue (Ch 17-18):
+- Same structure
+
+## Testing & Quality Issues
+For each issue (Ch 9, 15):
+- Same structure
+
+## Recommendations
+Priority-ordered from most critical to nice-to-have.
+Each recommendation references the specific chapter/concept.
+```
+
+### Common Web Scraping Anti-Patterns to Flag
+
+- **No error handling on requests** → Ch 1, 14: Wrap requests in try/except; handle ConnectionError, Timeout, HTTPError
+- **Hardcoded selectors without fallbacks** → Ch 2: Use multiple selector strategies; check for None before accessing attributes
+- **No rate limiting** → Ch 14: Add time.sleep() between requests; respect server resources
+- **Missing User-Agent header** → Ch 14: Set a descriptive User-Agent; rotate if needed for scale
+- **Not using sessions** → Ch 10: Use requests.Session() for cookie persistence and connection pooling
+- **Ignoring robots.txt** → Ch 18: Parse and respect robots.txt before crawling
+- **No URL deduplication** → Ch 3: Track visited URLs in a set; normalize URLs before comparing
+- **Using regex to parse HTML** → Ch 2: Use BeautifulSoup or lxml, not regex, for HTML parsing
+- **Not handling JavaScript content** → Ch 11: If data loads via Ajax, use Selenium or find the underlying API
+- **Storing data without validation** → Ch 6, 8: Validate and clean data before storage; handle encoding
+- **No logging** → Ch 5: Log requests, responses, errors, extracted items; track progress
+- **Sequential when parallel is needed** → Ch 16: Use threading/multiprocessing for large-scale scraping
+- **Ignoring encoding issues** → Ch 7, 8: Handle UTF-8, detect encoding, normalize Unicode
+- **No tests for parsers** → Ch 15: Write unit tests with saved HTML fixtures; test selector robustness
+- **Credentials in code** → Ch 10: Use environment variables or config files for login credentials
+- **Not storing raw responses** → Ch 6: Save raw HTML for re-parsing; don't rely only on extracted data
+
+---
+
+## General Guidelines
+
+- **BeautifulSoup for simple scraping, Scrapy for scale** — Match the tool to the complexity
+- **Check for APIs first** — Many sites have APIs (documented or undocumented) that are easier than scraping
+- **Respect the site** — Rate limit, identify yourself, follow robots.txt, check ToS
+- **Parse defensively** — HTML structure changes; always handle missing elements gracefully
+- **Test with saved pages** — Save HTML fixtures and test parsers offline; reduces requests and enables CI
+- **Clean data early** — Normalize strings, handle encoding, strip whitespace at extraction time
+- For deeper practice details, read `references/practices-catalog.md` before building scrapers.
+- For review checklists, read `references/review-checklist.md` before reviewing scrapers.

package/web-scraping-python/assets/example_asset.txt

@@ -0,0 +1 @@
+

package/web-scraping-python/references/api_reference.md

@@ -0,0 +1,393 @@
+# Web Scraping with Python — Practices Catalog
+
+Chapter-by-chapter catalog of practices from *Web Scraping with Python*
+by Ryan Mitchell for scraper building.
+
+---
+
+## Chapter 1: Your First Web Scraper
+
+### Basic Fetching
+- **urllib.request** — `urlopen(url)` returns an HTTPResponse object; read `.read()` for HTML bytes
+- **requests library** — Preferred over urllib; `requests.get(url)` with headers, params, timeout support
+- **Error handling** — Catch `HTTPError` (4xx/5xx), `URLError` (server not found), and connection timeouts
+- **Response checking** — Always check `response.status_code`; handle 403 (forbidden), 404 (not found), 500 (server error)
+
+### BeautifulSoup Basics
+- **Creating soup** — `BeautifulSoup(html, 'html.parser')` or use `'lxml'` for speed
+- **Direct tag access** — `soup.h1`, `soup.title` returns first matching tag
+- **Tag attributes** — `tag.attrs` returns dict; `tag['href']` for specific attribute; `tag.get_text()` for text content
+- **None checking** — Always check if `soup.find()` returns None before accessing attributes
+
+---
+
+## Chapter 2: Advanced HTML Parsing
+
+### find and findAll
+- **`find(tag, attributes, recursive, text, keywords)`** — Returns first matching element
+- **`findAll(tag, attributes, recursive, text, limit, keywords)`** — Returns list of all matches
+- **Attribute filtering** — `find('div', {'class': 'price'})`, `find('span', {'id': 'result'})`
+- **Multiple tags** — `findAll(['h1', 'h2', 'h3'])` matches any of the listed tags
+- **Text search** — `findAll(text='exact match')` or `findAll(text=re.compile('pattern'))`
+
+### CSS Selectors
+- **`select(selector)`** — Use CSS selectors: `soup.select('div.content > p')`, `soup.select('#main .item')`
+- **Common selectors** — `tag`, `.class`, `#id`, `tag.class`, `parent > child`, `ancestor descendant`, `tag[attr=val]`
+- **Pseudo-selectors** — `:nth-of-type()`, `:first-child`, etc. for positional selection
+
+### Navigating the DOM Tree
+- **Children** — `tag.children` (direct children iterator), `tag.descendants` (all descendants)
+- **Siblings** — `tag.next_sibling`, `tag.previous_sibling`, `tag.next_siblings` (iterator)
+- **Parents** — `tag.parent`, `tag.parents` (iterator up to document root)
+- **Navigation tip** — NavigableString objects (text nodes) count as siblings; use `.find_next_sibling('tag')` to skip
+
+### Regular Expressions with BeautifulSoup
+- **Regex in find** — `soup.find('img', {'src': re.compile(r'\.jpg$')})` matches pattern against attribute
+- **Regex in findAll** — `soup.findAll('a', {'href': re.compile(r'^/wiki/')})` for link patterns
+- **Text regex** — `soup.findAll(text=re.compile(r'\$[\d,]+'))` for finding price patterns
+
+### Lambda Functions
+- **Lambda filters** — `soup.find_all(lambda tag: len(tag.attrs) == 2)` for custom tag filtering
+- **Complex conditions** — Combine tag name, attributes, text content in lambda for precise selection
+
+---
+
+## Chapter 3: Writing Web Crawlers
+
+### Single-Domain Crawling
+- **Internal link collection** — Find all `<a>` tags; filter for same-domain links using `urlparse`
+- **URL normalization** — Resolve relative URLs with `urljoin`; strip fragments and query strings for dedup
+- **Visited tracking** — Maintain a `set()` of visited URLs; check before fetching
+- **Breadth-first** — Use a queue (collections.deque) for BFS traversal of site
+- **Depth-first** — Use a stack (list) for DFS; useful for deep hierarchical sites
+
+### Building Robust Crawlers
+- **Recursive crawling** — Function that fetches page, extracts links, recurses on unvisited links
+- **Data extraction during crawl** — Extract target data while crawling; don't just collect URLs
+- **Depth limiting** — Set maximum crawl depth to prevent infinite recursion
+- **URL deduplication** — Normalize URLs before adding to visited set; handle trailing slashes, www prefix
+
+---
+
+## Chapter 4: Web Crawling Models
+
+### Planning a Crawl
+- **Site mapping** — Understand site structure before coding; identify URL patterns, pagination, categories
+- **Crawl scope** — Define which pages/sections to include or exclude
+- **Data schema** — Define what to extract before building; normalize across different page layouts
+
+### Handling Different Layouts
+- **Template detection** — Sites may use different templates for different content types
+- **Conditional parsing** — Check page type (product vs category vs article) and apply appropriate parser
+- **Data normalization** — Map different field names/formats from different layouts to a unified schema
+
+### Cross-Site Crawling
+- **Multi-domain** — Maintain per-domain settings (delays, selectors, credentials)
+- **Link following policies** — Decide which external links to follow; whitelist/blacklist domains
+- **Politeness per domain** — Track per-domain request timing; respect each site's robots.txt
+
+---
+
+## Chapter 5: Scrapy
+
+### Scrapy Architecture
+- **Spider** — Defines how to crawl and parse; subclass `scrapy.Spider`; implement `parse()` method
+- **Items** — Structured data containers; define fields with `scrapy.Item` and `scrapy.Field()`
+- **Pipelines** — Process items after extraction; validate, clean, store to database/file
+- **Middleware** — Hook into request/response processing; add headers, proxy rotation, retry logic
+- **Settings** — Configure concurrency (`CONCURRENT_REQUESTS`), delays (`DOWNLOAD_DELAY`), user agent, etc.
+
+### CrawlSpider
+- **Rules** — Define `Rule(LinkExtractor(...), callback=...)` for automatic link following
+- **LinkExtractor** — Filter links by `allow` (regex), `deny`, `restrict_css`, `restrict_xpaths`
+- **Callback** — Assign parse methods to different URL patterns; `follow=True` for recursive crawling
+
+### Scrapy Best Practices
+- **Item loaders** — Use `ItemLoader` for cleaner extraction with input/output processors
+- **Logging** — Configure log levels (`LOG_LEVEL = 'INFO'`); log to file for production runs
+- **Autothrottle** — Enable `AUTOTHROTTLE_ENABLED` for adaptive request pacing
+- **Feed exports** — Built-in export to JSON, CSV, XML via `-o output.json`
+- **Contracts** — Add docstring-based contracts for spider testing
+
+---
+
+## Chapter 6: Storing Data
+
+### File Storage
+- **CSV** — Use `csv.writer` or `csv.DictWriter`; handle encoding with `encoding='utf-8'`
+- **JSON** — Use `json.dump()` for structured data; JSON Lines for streaming/appending
+- **Raw files** — Download images, PDFs with `urllib.request.urlretrieve()` or `requests.get()` with streaming
+
+### Database Storage
+- **MySQL** — Use `pymysql` connector; parameterized queries to prevent SQL injection
+- **PostgreSQL** — Use `psycopg2`; connection pooling for concurrent scrapers
+- **SQLite** — Use built-in `sqlite3` for lightweight local storage; good for prototyping
+- **Schema design** — Design tables to match extracted data; use appropriate types; add indexes on lookup columns
+
+### Email Integration
+- **smtplib** — Send scraped data or alerts via email; useful for monitoring scraper results
+- **Notifications** — Alert on scraper failures, unusual data patterns, or completion
+
+### Storage Best Practices
+- **Idempotent storage** — Check for duplicates before inserting; use UPSERT patterns
+- **Raw preservation** — Store raw HTML alongside extracted data for re-parsing capability
+- **Batch operations** — Use bulk inserts for efficiency; commit in batches, not per-row
+- **Connection management** — Use context managers; close connections properly; handle reconnection
+
+---
+
+## Chapter 7: Reading Documents
+
+### PDF Extraction
+- **PDFMiner** — Extract text from PDFs; handle multi-column layouts and tables
+- **Page-by-page** — Process PDFs page by page for memory efficiency
+- **Tables in PDFs** — Use tabula-py or camelot for structured table extraction
+
+### Word Documents
+- **python-docx** — Read `.docx` files; extract paragraphs, tables, headers
+- **Older formats** — Handle `.doc` files with antiword or textract
+
+### Encoding
+- **Character detection** — Use `chardet` to detect file encoding when unknown
+- **UTF-8 normalization** — Convert all text to UTF-8; handle BOM (Byte Order Mark)
+- **HTML encoding** — Read `<meta charset>` tag; handle entity references (`&amp;`, `&lt;`)
+
+---
+
+## Chapter 8: Cleaning Dirty Data
+
+### String Normalization
+- **Whitespace** — Strip leading/trailing whitespace; normalize internal whitespace (multiple spaces to one)
+- **Unicode normalization** — Use `unicodedata.normalize('NFKD', text)` for consistent Unicode representation
+- **Case normalization** — Lowercase for comparison; preserve original for display
+
+### Regex Cleaning
+- **Pattern extraction** — Use regex groups to extract structured data from messy text (prices, dates, phone numbers)
+- **Substitution** — `re.sub()` to remove or replace unwanted characters and patterns
+- **Compiled patterns** — Pre-compile frequently used patterns with `re.compile()` for performance
+
+### Data Normalization
+- **Date formats** — Parse various date formats with `dateutil.parser`; store in ISO 8601
+- **Number formats** — Handle commas, currency symbols, percentage signs; convert to numeric types
+- **Address normalization** — Standardize address components; handle abbreviations
+
+### OpenRefine
+- **Faceting** — Group similar values to find inconsistencies
+- **Clustering** — Automatically find and merge similar values (fingerprint, n-gram, etc.)
+- **GREL expressions** — Transform data with OpenRefine's expression language
+
+---
+
+## Chapter 9: Natural Language Processing
+
+### Text Analysis
+- **N-grams** — Extract sequences of N words; useful for finding common phrases and patterns
+- **Frequency analysis** — Count word/phrase frequencies; identify key topics in scraped text
+- **Stop words** — Filter common words (the, is, at) to focus on meaningful content
+
+### Markov Models
+- **Text generation** — Build Markov chains from scraped text; generate similar-style text
+- **Chain order** — Higher order (2-gram, 3-gram) produces more coherent but less varied output
+
+### NLTK
+- **Tokenization** — Split text into words and sentences with NLTK tokenizers
+- **Part-of-speech tagging** — Tag words as nouns, verbs, etc. for structured extraction
+- **Named entity recognition** — Extract names, organizations, locations from text
+- **Stemming/lemmatization** — Reduce words to base forms for better matching and analysis
+
+---
+
+## Chapter 10: Crawling Through Forms and Logins
+
+### Form Submission
+- **POST requests** — `requests.post(url, data={'field': 'value'})` for form submission
+- **CSRF tokens** — Extract hidden CSRF token from form HTML; include in POST data
+- **Form fields** — Inspect form with browser DevTools; identify all required fields including hidden ones
+- **File uploads** — Use `files` parameter in `requests.post()` for multipart form data
+
+### Session Management
+- **requests.Session()** — Maintains cookies across requests; handles redirects; connection pooling
+- **Cookie persistence** — Session object automatically stores and sends cookies
+- **Login flow** — GET login page → extract CSRF → POST credentials → use session for authenticated pages
+
+### Authentication
+- **HTTP Basic Auth** — `requests.get(url, auth=('user', 'pass'))` for Basic authentication
+- **Token-based** — Extract auth token from login response; send in headers for subsequent requests
+- **OAuth** — Use `requests-oauthlib` for OAuth-protected APIs
+- **Session expiry** — Detect expired sessions (redirects to login); re-authenticate automatically
+
+---
+
+## Chapter 11: Scraping JavaScript
+
+### Selenium WebDriver
+- **Setup** — `webdriver.Chrome()` or `webdriver.Firefox()`; requires matching driver binary
+- **Headless mode** — `options.add_argument('--headless')` for browser without GUI; essential for servers
+- **Navigation** — `driver.get(url)`; `driver.find_element(By.CSS_SELECTOR, selector)`
+- **Interaction** — `.click()`, `.send_keys()`, `.clear()` on elements; simulate user behavior
+
+### Waiting for Content
+- **Implicit waits** — `driver.implicitly_wait(10)` sets default wait for element finding
+- **Explicit waits** — `WebDriverWait(driver, 10).until(EC.presence_of_element_located(...))` for specific conditions
+- **Expected conditions** — `element_to_be_clickable`, `visibility_of_element_located`, `text_to_be_present_in_element`
+- **Custom waits** — Write lambda conditions for complex wait scenarios
+
+### JavaScript Execution
+- **Execute script** — `driver.execute_script('return document.title')` runs JS in page context
+- **Scroll page** — `driver.execute_script('window.scrollTo(0, document.body.scrollHeight)')` for infinite scroll
+- **Extract data** — Execute JS to extract data from page variables, localStorage, or DOM
+
+### Ajax Handling
+- **Wait for Ajax** — Wait for specific elements that load asynchronously
+- **Network monitoring** — Intercept XHR requests to find underlying API endpoints
+- **Alternative approach** — If you can identify the API endpoint, use `requests` directly instead of Selenium
+
+---
+
+## Chapter 12: Crawling Through APIs
+
+### REST API Basics
+- **HTTP methods** — GET (read), POST (create), PUT (update), DELETE (remove)
+- **JSON responses** — `response.json()` for parsing; handle nested objects and arrays
+- **Headers** — Set `Accept: application/json`, `Authorization: Bearer token`
+- **Query parameters** — `requests.get(url, params={'key': 'value'})` for clean URL building
+
+### Undocumented APIs
+- **Browser DevTools** — Use Network tab to discover API calls made by JavaScript
+- **XHR filtering** — Filter network requests to XHR/Fetch to find data endpoints
+- **Request replication** — Copy request headers, cookies, parameters from DevTools to Python
+- **API reverse engineering** — Study request patterns to understand pagination, filtering, authentication
+
+### API Best Practices
+- **Rate limiting** — Respect rate limit headers; implement backoff on 429 responses
+- **Pagination** — Handle cursor-based, offset-based, and link-header pagination
+- **Error handling** — Retry on 5xx errors with exponential backoff; don't retry on 4xx
+- **Authentication** — Store API keys securely; handle token refresh for OAuth
+
+---
+
+## Chapter 13: Image Processing and OCR
+
+### Pillow (PIL)
+- **Image loading** — `Image.open(path)` or from URL response content
+- **Manipulation** — Resize, crop, rotate, filter for preprocessing before OCR
+- **Thresholding** — Convert to grayscale; apply threshold for clean black/white text
+
+### Tesseract OCR
+- **pytesseract** — `pytesseract.image_to_string(image)` for text extraction from images
+- **Preprocessing** — Clean images before OCR: denoise, deskew, threshold, resize
+- **Language support** — Specify language with `lang='eng'`; install language packs as needed
+- **Confidence** — Use `image_to_data()` for per-word confidence scores; filter low confidence
+
+### CAPTCHA Handling
+- **Simple CAPTCHAs** — Preprocessing + OCR may solve simple text CAPTCHAs
+- **Complex CAPTCHAs** — Consider CAPTCHA-solving services or rethink approach (use API instead)
+- **Ethical note** — CAPTCHAs exist to prevent automated access; respect their purpose
+
+---
+
+## Chapter 14: Avoiding Scraping Traps
+
+### Headers and Identity
+- **User-Agent** — Set a realistic browser User-Agent string; rotate for large-scale scraping
+- **Accept headers** — Include Accept, Accept-Language, Accept-Encoding to mimic real browsers
+- **Referer** — Set appropriate Referer header when navigating between pages
+- **Cookie handling** — Accept and send cookies; use sessions for automatic management
+
+### Behavioral Patterns
+- **Request timing** — Add random delays between requests (1-5 seconds); avoid perfectly regular intervals
+- **Navigation patterns** — Don't jump straight to data pages; mimic human browsing (home → category → product)
+- **Click patterns** — With Selenium, click through pages naturally rather than jumping directly to URLs
+
+### Honeypot Detection
+- **Hidden links** — Check for CSS `display:none` or `visibility:hidden` links; avoid following them
+- **Hidden form fields** — Pre-filled hidden fields may be traps; don't submit unexpected values
+- **Link patterns** — Suspicious URL patterns or link text may indicate honeypots
+
+### IP and Session Management
+- **Proxy rotation** — Rotate IP addresses for large-scale scraping; use proxy services
+- **Session rotation** — Create new sessions periodically; don't use same cookies indefinitely
+- **Fingerprint diversity** — Vary headers, timing, and behavior to avoid fingerprinting
+
+---
+
+## Chapter 15: Testing Scrapers
+
+### Unit Testing
+- **Parse function tests** — Test parsing functions with saved HTML files; verify extracted data
+- **Fixture files** — Save representative HTML pages as test fixtures; don't hit live sites in tests
+- **Edge cases** — Test with missing elements, empty pages, different layouts, malformed HTML
+
+### Integration Testing
+- **End-to-end** — Test full scrape pipeline from fetch to storage with known target pages
+- **Selenium tests** — Use Selenium for testing JavaScript-heavy scraping flows
+- **Mock responses** — Use `responses` or `requests-mock` libraries for HTTP mocking in tests
+
+### Testing Best Practices
+- **Site change detection** — Periodically check if site structure has changed; alert on selector failures
+- **Regression testing** — Compare current results against known-good baselines
+- **CI integration** — Run scraper tests in CI pipeline; catch issues before deployment
+
+---
+
+## Chapter 16: Parallel Web Scraping
+
+### Threading
+- **threading module** — Use for I/O-bound scraping; GIL doesn't block network operations
+- **Thread pool** — `concurrent.futures.ThreadPoolExecutor` for managed thread pools
+- **Thread safety** — Use locks for shared state (counters, result lists); prefer queues for task distribution
+
+### Multiprocessing
+- **multiprocessing module** — Use for CPU-bound processing (parsing, cleaning); bypasses GIL
+- **Process pool** — `concurrent.futures.ProcessPoolExecutor` for managed process pools
+- **Inter-process communication** — Use Queue for task distribution; Pipe for point-to-point
+
+### Queue-Based Architecture
+- **Producer-consumer** — Producer adds URLs to queue; consumers fetch and parse in parallel
+- **URL frontier** — Priority queue for managing which URLs to crawl next
+- **Result aggregation** — Collect results from workers into shared storage
+
+### Parallel Best Practices
+- **Per-domain limits** — Limit concurrent requests per domain even with parallel scraping
+- **Graceful shutdown** — Handle KeyboardInterrupt; drain queues cleanly on shutdown
+- **Error isolation** — One worker's failure shouldn't crash the entire scraping operation
+- **Progress tracking** — Log completed/remaining tasks; monitor worker health
+
+---
+
+## Chapter 17: Remote Scraping
+
+### Tor
+- **Tor proxy** — Route requests through Tor network for anonymity; `socks5://127.0.0.1:9150`
+- **IP verification** — Check IP with a service like httpbin.org/ip to verify Tor is active
+- **Performance** — Tor is slow; use only when anonymity is required
+- **Circuit rotation** — Signal Tor to create new circuit for fresh IP; don't rotate too frequently
+
+### Proxy Services
+- **Rotating proxies** — Commercial proxy services provide rotating IP pools
+- **Proxy types** — HTTP/HTTPS proxies, SOCKS proxies; understand the difference
+- **Proxy configuration** — `requests.get(url, proxies={'http': proxy_url})`; or configure in Scrapy settings
+
+### Cloud-Based Scraping
+- **Headless instances** — Run scrapers on cloud VMs (AWS, GCP, DigitalOcean) for scale
+- **Containerization** — Docker containers for consistent scraper environments
+- **Scheduling** — Use cron, cloud schedulers, or orchestration tools for recurring scrapes
+- **Cost management** — Right-size instances; use spot/preemptible instances for batch scraping
+
+---
+
+## Chapter 18: Legalities and Ethics
+
+### Legal Framework
+- **robots.txt** — Machine-readable file at `/robots.txt`; specifies which paths are allowed/disallowed
+- **Terms of Service** — Many sites prohibit scraping in ToS; understand the legal weight
+- **CFAA** — Computer Fraud and Abuse Act (US); accessing computers "without authorization" is a federal crime
+- **Copyright** — Scraped data may be copyrighted; fair use depends on purpose and amount
+- **GDPR** — If scraping personal data of EU citizens, GDPR obligations apply
+
+### Ethical Scraping
+- **Respect the site** — Don't overload servers; honor rate limits; scrape during off-peak hours
+- **Identify yourself** — Use a descriptive User-Agent; provide contact email for site administrators
+- **Minimize footprint** — Only scrape what you need; don't archive entire sites unnecessarily
+- **Data handling** — Handle scraped personal data responsibly; minimize collection and storage
+- **Give back** — If possible, contribute to the site or community; don't just extract value