@hanzlaa/rcode 4.3.3 → 4.4.0

package/rcode/skills/seo/technical-seo-checker/references/http-status-codes.md

@@ -0,0 +1,705 @@
+# HTTP Status Codes for SEO
+
+SEO-relevant HTTP status codes, their implications, and how to diagnose and fix issues.
+
+## Status Code Categories
+
+- **2xx**: Success - Request succeeded
+- **3xx**: Redirection - Further action needed
+- **4xx**: Client Error - Problem with the request
+- **5xx**: Server Error - Server failed to fulfill request
+
+---
+
+## 2xx Success Codes
+
+### 200 OK
+
+**What it means**: Request succeeded, content returned normally.
+
+**SEO impact**: Positive - page is accessible and indexable.
+
+**When to use**: Standard response for all working pages.
+
+**When it's a problem**: When different URLs return 200 for same content (should use 301 redirect).
+
+---
+
+### 204 No Content
+
+**What it means**: Request succeeded but no content to return.
+
+**SEO impact**: Neutral - rarely used for pages meant to be indexed.
+
+**Common use**: API responses, AJAX requests.
+
+---
+
+## 3xx Redirection Codes
+
+### 301 Moved Permanently
+
+**What it means**: Resource permanently moved to new URL. All link equity transfers.
+
+**SEO impact**: Positive when used correctly - passes 90-99% of link equity.
+
+**When to use**:
+- Permanently changing URL structure
+- Consolidating duplicate content
+- Moving to new domain
+- Changing HTTP to HTTPS
+- Changing www to non-www (or vice versa)
+
+**Example header**:
+```
+HTTP/1.1 301 Moved Permanently
+Location: https://example.com/new-page
+```
+
+**Common mistakes**:
+- Using 302 instead of 301 for permanent changes
+- Creating redirect chains (A→B→C)
+- Redirecting to irrelevant pages
+- Not redirecting HTTP to HTTPS
+
+**How to implement**:
+- **.htaccess** (Apache): `Redirect 301 /old-page /new-page`
+- **nginx**: `rewrite ^/old-page$ /new-page permanent;`
+- **Server-side**: Set Location header with 301 status
+
+---
+
+### 302 Found (Temporary Redirect)
+
+**What it means**: Resource temporarily at different URL. Original URL should still be used.
+
+**SEO impact**: Neutral to negative - does NOT pass full link equity. Search engines keep indexing original URL.
+
+**When to use**:
+- A/B testing
+- Temporary promotions
+- Maintenance redirects
+- Geolocation redirects (sometimes)
+
+**When NOT to use**: Permanent URL changes (use 301).
+
+**Warning**: Google may treat long-standing 302s as 301s, but better to be explicit.
+
+---
+
+### 303 See Other
+
+**What it means**: Response can be found at another URI using GET.
+
+**SEO impact**: Minimal - rarely used for SEO purposes.
+
+**Common use**: After form submissions, redirect to results page.
+
+---
+
+### 307 Temporary Redirect
+
+**What it means**: Temporary redirect that preserves request method (POST stays POST).
+
+**SEO impact**: Similar to 302 - temporary, doesn't pass full link equity.
+
+**Difference from 302**: Guarantees request method won't change (more precise than 302).
+
+**When to use**: Temporary redirects where HTTP method preservation matters.
+
+---
+
+### 308 Permanent Redirect
+
+**What it means**: Permanent redirect that preserves request method.
+
+**SEO impact**: Similar to 301 - passes link equity.
+
+**Difference from 301**: Guarantees request method won't change (POST stays POST).
+
+**When to use**: Permanent redirects where method preservation matters (rare for SEO).
+
+---
+
+### Redirect Chain Issues
+
+**Problem**: Multiple redirects before reaching final destination.
+
+**Example chain**:
+```
+http://example.com/page
+  → https://example.com/page (redirect 1)
+  → https://www.example.com/page (redirect 2)
+  → https://www.example.com/new-page (redirect 3)
+```
+
+**SEO impact**:
+- Slows page load (each redirect = new HTTP request)
+- Dilutes link equity with each hop
+- Wastes crawl budget
+- Poor user experience
+
+**How to fix**: Redirect directly from original URL to final destination.
+
+**Fixed version**:
+```
+http://example.com/page
+  → https://www.example.com/new-page (single redirect)
+```
+
+---
+
+### Redirect Loops
+
+**Problem**: Redirects create infinite loop.
+
+**Example**:
+```
+/page-a → /page-b
+/page-b → /page-a
+```
+
+**SEO impact**: Severe - page completely inaccessible.
+
+**Symptoms**:
+- Browser shows "Too many redirects" error
+- Page never loads
+- Search Console shows crawl errors
+
+**How to diagnose**:
+1. Use redirect checker tool
+2. Check .htaccess or nginx config for conflicting rules
+3. Review server-side redirect logic
+
+**How to fix**:
+1. Identify conflicting redirect rules
+2. Remove or correct the loop
+3. Test thoroughly
+4. Request recrawl in Search Console
+
+---
+
+## 4xx Client Error Codes
+
+### 404 Not Found
+
+**What it means**: Requested resource doesn't exist.
+
+**SEO impact**: Neutral to negative depending on context.
+
+**When 404s are OK**:
+- Legitimately deleted pages with no equivalent
+- Never-existed URLs from typos
+- Temporary content that expired (old promotions)
+- Intentionally removed low-quality content
+
+**When 404s are problems**:
+- Pages that should exist are returning 404
+- Previously working pages now broken
+- Important pages missing from navigation
+- High-traffic pages deleted without redirect
+
+**How to fix**:
+1. **If content moved**: Set up 301 redirect to new location
+2. **If content deleted**: Either keep 404 or redirect to relevant category
+3. **If never existed**: Leave as 404
+4. **If important**: Restore the page
+
+**Monitoring 404s**:
+- Check Search Console → Coverage → Not found (404)
+- Review referrer data to see what's linking to 404s
+- Fix high-value 404s first (most traffic/backlinks)
+
+**Soft 404s** (BAD):
+- Page returns 200 but shows "not found" message
+- Search engines may keep page indexed
+- Creates duplicate content issues
+- Fix: Return proper 404 status code
+
+---
+
+### 410 Gone
+
+**What it means**: Resource permanently deleted, never coming back.
+
+**SEO impact**: Stronger signal than 404 - tells search engines not to return.
+
+**When to use**:
+- Discontinued products
+- Expired promotions
+- Permanently removed content
+- Outdated information
+
+**Difference from 404**:
+- 404: "Not found" (might exist at another URL)
+- 410: "Gone forever" (don't look for it)
+
+**When to use 410 vs 301**:
+- Use 410: No equivalent replacement exists
+- Use 301: Relevant alternative exists
+
+**How search engines respond**:
+- Faster de-indexing than 404
+- Stop crawling sooner
+- Better for crawl budget
+
+---
+
+### 403 Forbidden
+
+**What it means**: Server understood request but refuses to authorize it.
+
+**SEO impact**: Negative - page inaccessible and won't be indexed.
+
+**Common causes**:
+- Permission restrictions
+- IP blocking
+- .htaccess restrictions
+- File permissions (chmod)
+- Authentication required
+
+**When it's intentional**:
+- Admin areas
+- Member-only content
+- Geographic restrictions
+
+**When it's a problem**:
+- Public pages returning 403
+- Search engine bots blocked
+- Accidental permission changes
+
+**How to diagnose**:
+1. Check .htaccess for IP restrictions
+2. Verify file permissions (should be 644 for files, 755 for directories)
+3. Check server-level access rules
+4. Test with different IPs/user-agents
+
+**How to fix**:
+1. Adjust file permissions: `chmod 644 filename`
+2. Remove blocking rules from .htaccess
+3. Whitelist search engine bots
+4. Review server firewall rules
+
+---
+
+### 401 Unauthorized
+
+**What it means**: Authentication required but not provided or failed.
+
+**SEO impact**: Negative - page won't be indexed.
+
+**Common causes**:
+- Password-protected pages
+- HTTP Basic Authentication
+- Expired sessions
+- Missing credentials
+
+**When it's intentional**: Member areas, staging sites, admin panels.
+
+**How to handle for SEO**:
+- Don't password-protect pages you want indexed
+- Use separate staging domain with 401
+- For members-only content, show teaser with meta robots noindex
+
+---
+
+### 429 Too Many Requests
+
+**What it means**: User/bot sent too many requests in given timeframe (rate limiting).
+
+**SEO impact**: Negative if search engines can't crawl.
+
+**Common causes**:
+- Aggressive crawling
+- DDoS protection triggered
+- API rate limits
+- Server throttling
+
+**How to handle**:
+1. Check Googlebot isn't being rate-limited (use Search Console)
+2. Whitelist verified search engine bots
+3. Configure rate limits appropriately
+4. Monitor crawl rate in Search Console
+
+---
+
+## 5xx Server Error Codes
+
+### 500 Internal Server Error
+
+**What it means**: Generic server error, something went wrong.
+
+**SEO impact**: Very negative if persistent - prevents indexing and ranking.
+
+**Common causes**:
+- PHP/code errors
+- Database connection issues
+- .htaccess syntax errors
+- Resource limits exceeded
+- Plugin/theme conflicts (WordPress)
+
+**How to diagnose**:
+1. Check server error logs
+2. Review recent code/config changes
+3. Test locally or on staging
+4. Disable plugins one by one (if CMS)
+5. Check .htaccess syntax
+
+**How to fix**:
+1. Review error logs for specific error
+2. Roll back recent changes
+3. Fix code errors
+4. Increase resource limits if needed
+5. Test thoroughly before re-deploying
+
+**Monitoring**: Set up alerts for 500 errors (sudden spike = problem).
+
+---
+
+### 502 Bad Gateway
+
+**What it means**: Server received invalid response from upstream server.
+
+**SEO impact**: Negative if persistent - prevents crawling/indexing.
+
+**Common causes**:
+- Proxy/load balancer issues
+- Upstream server down
+- Timeout issues
+- Firewall blocking
+
+**Common scenarios**:
+- CDN can't reach origin server
+- Application server crashed
+- Database server unresponsive
+
+**How to fix**:
+1. Check upstream server status
+2. Verify firewall rules
+3. Check timeout settings
+4. Restart proxy/load balancer if needed
+5. Review CDN configuration
+
+---
+
+### 503 Service Unavailable
+
+**What it means**: Server temporarily unable to handle request.
+
+**SEO impact**: Neutral if truly temporary with Retry-After header. Negative if prolonged.
+
+**Common causes**:
+- Maintenance mode
+- Server overload
+- Database down
+- Resource exhaustion
+
+**Proper use for maintenance**:
+```
+HTTP/1.1 503 Service Unavailable
+Retry-After: 3600
+```
+
+**Best practices for maintenance**:
+1. Use 503 (not 404 or 500)
+2. Include Retry-After header
+3. Keep maintenance brief (<24 hours)
+4. Schedule during low-traffic times
+5. Inform users with clear message
+
+**How search engines handle 503**:
+- Short-term (hours): Will retry, no ranking impact
+- Long-term (days+): May drop rankings, de-index pages
+
+---
+
+### 504 Gateway Timeout
+
+**What it means**: Server didn't receive timely response from upstream server.
+
+**SEO impact**: Negative - prevents crawling.
+
+**Common causes**:
+- Slow database queries
+- External API timeouts
+- Insufficient server resources
+- Network issues
+
+**How to fix**:
+1. Optimize slow queries
+2. Increase timeout limits
+3. Add caching
+4. Scale server resources
+5. Review external dependencies
+
+---
+
+## Status Code Decision Flowchart
+
+### Content Moved Permanently?
+→ YES: Use **301 redirect**
+→ NO: Continue
+
+### Content Moved Temporarily?
+→ YES: Use **302 redirect**
+→ NO: Continue
+
+### Content Deleted with No Replacement?
+→ YES: Use **404** (or **410** if permanently gone)
+→ NO: Continue
+
+### Content Exists at This URL?
+→ YES: Use **200 OK**
+→ NO: Use **404**
+
+### Need Authentication?
+→ YES: Use **401**
+→ NO: Continue
+
+### Access Forbidden?
+→ YES: Use **403**
+→ NO: Continue
+
+### Server Error?
+→ YES: Use **500**, **502**, **503**, or **504** depending on cause
+→ NO: Use **200 OK**
+
+---
+
+## Diagnosing Status Code Issues
+
+### Tools
+
+**Browser DevTools**:
+1. Open DevTools (F12)
+2. Go to Network tab
+3. Reload page
+4. Check status code in first request
+
+**cURL command**:
+```bash
+curl -I https://example.com/page
+```
+
+**Online checkers**:
+- httpstatus.io
+- redirect-checker.org
+- websiteplanet.com/webtools/redirects/
+
+**Google Search Console**:
+- Coverage report → Error/Excluded sections
+- URL Inspection tool → Check specific URLs
+
+---
+
+### Common Diagnostic Scenarios
+
+### "Page Won't Index"
+
+**Check**:
+1. Status code (should be 200)
+2. Redirects (shouldn't redirect away)
+3. 4xx/5xx errors
+4. robots.txt blocking
+5. noindex meta tag
+
+### "Page Disappeared from Results"
+
+**Check**:
+1. Returns 404/410/5xx
+2. Redirecting elsewhere (301/302)
+3. Changed to 403/401
+4. Server timing out (504)
+
+### "Traffic Dropped After Migration"
+
+**Check**:
+1. Old URLs return 404 (should be 301)
+2. Redirect chains (should be direct)
+3. Redirect loops
+4. Wrong redirect type (302 vs 301)
+5. Incorrect redirect targets
+
+---
+
+## Status Codes and Crawl Budget
+
+### Impact on Crawl Budget
+
+**Efficient (minimal impact)**:
+- 200 OK
+- 301 redirects (if minimal chains)
+- 410 Gone (removes from crawl queue)
+
+**Moderate impact**:
+- 302 redirects (search engine may keep checking)
+- 404 errors (search engines periodically recheck)
+- Redirect chains (multiple requests per URL)
+
+**High impact (wasteful)**:
+- 5xx errors (search engines retry frequently)
+- Redirect loops (waste crawl budget)
+- Soft 404s (search engine confused, keeps crawling)
+- 429 rate limiting (prevents efficient crawling)
+
+---
+
+## SEO Status Code Best Practices
+
+### For Migrations
+
+- [ ] Use 301 redirects for all permanently moved pages
+- [ ] Redirect directly to final destination (no chains)
+- [ ] Test all redirects before launching
+- [ ] Keep redirects in place for at least 1 year
+- [ ] Monitor 404 errors in Search Console post-launch
+- [ ] Map 1:1 where possible (old URL → equivalent new URL)
+
+### For Deleted Content
+
+- [ ] Use 301 if relevant replacement exists
+- [ ] Use 404 if no replacement and might return
+- [ ] Use 410 if permanently gone, never returning
+- [ ] Don't redirect to irrelevant pages (creates soft 404)
+- [ ] Create custom 404 page with search and navigation
+
+### For Maintenance
+
+- [ ] Use 503 with Retry-After header
+- [ ] Keep maintenance window brief (<24 hours)
+- [ ] Create user-friendly maintenance page
+- [ ] Inform users of expected downtime
+- [ ] Monitor Search Console for crawl issues
+
+### For Performance
+
+- [ ] Minimize redirect chains
+- [ ] Fix redirect loops immediately
+- [ ] Monitor 5xx errors closely
+- [ ] Set up alerts for sudden status code changes
+- [ ] Optimize to reduce 504 timeouts
+
+---
+
+## Status Code Monitoring
+
+### Key Metrics to Track
+
+**In Search Console**:
+- Crawl errors by type
+- Server errors (5xx) trend
+- Not found (404) trend
+- Redirect errors
+
+**In analytics**:
+- 404 page views
+- Entry pages with high exit rate (might be errors)
+- Sudden traffic drops (could indicate status code issues)
+
+**Server logs**:
+- Status code distribution
+- 5xx error frequency
+- Unusual patterns
+
+### Setting Up Alerts
+
+**Alert on**:
+- Sudden increase in 5xx errors
+- Increase in 404 errors
+- New redirect chains
+- Crawl error spikes in Search Console
+
+**Tools**:
+- Google Search Console email alerts
+- Server monitoring (UptimeRobot, Pingdom)
+- Log analysis tools
+- Custom scripts for log monitoring
+
+---
+
+## Quick Reference Table
+
+| Code | Name | SEO Impact | Use When | Passes Link Equity? |
+|------|------|------------|----------|---------------------|
+| 200 | OK | ✅ Positive | Page works normally | N/A (original URL) |
+| 301 | Moved Permanently | ✅ Positive | Permanent URL change | ✅ Yes (~90-99%) |
+| 302 | Found | ⚠️ Neutral | Temporary redirect | ❌ No |
+| 307 | Temporary Redirect | ⚠️ Neutral | Temporary (method preserved) | ❌ No |
+| 308 | Permanent Redirect | ✅ Positive | Permanent (method preserved) | ✅ Yes |
+| 404 | Not Found | ⚠️ Neutral | Content doesn't exist | N/A |
+| 410 | Gone | ⚠️ Neutral | Permanent deletion | N/A |
+| 403 | Forbidden | ❌ Negative | Access denied | N/A |
+| 401 | Unauthorized | ❌ Negative | Auth required | N/A |
+| 500 | Internal Server Error | ❌ Negative | Server error | N/A |
+| 502 | Bad Gateway | ❌ Negative | Upstream error | N/A |
+| 503 | Service Unavailable | ⚠️ Neutral | Temporary downtime | N/A |
+| 504 | Gateway Timeout | ❌ Negative | Timeout error | N/A |
+
+---
+
+## Status Code Testing Checklist
+
+Before launching site changes:
+
+- [ ] Test all redirects return correct status codes
+- [ ] Verify no redirect chains exist
+- [ ] Check no redirect loops present
+- [ ] Confirm important pages return 200
+- [ ] Ensure deleted pages return 404/410 (not 200)
+- [ ] Verify 301s point to correct destinations
+- [ ] Test with multiple user-agents
+- [ ] Check status codes in Search Console
+- [ ] Monitor server logs for unusual patterns
+- [ ] Set up alerts for error spikes
+
+---
+
+## Technical SEO Severity Framework
+
+### Issue Classification
+
+| Severity | Impact Description | Examples | Response Time |
+|----------|-------------------|---------|---------------|
+| **Critical** | Prevents indexation or causes site-wide issues | Robots.txt blocking site, noindex on key pages, site-wide 500 errors | Same day |
+| **High** | Significantly impacts rankings or user experience | Slow page speed, missing hreflang, duplicate content, redirect chains | Within 1 week |
+| **Medium** | Affects specific pages or has moderate impact | Missing schema, suboptimal canonicals, thin content pages | Within 1 month |
+| **Low** | Minor optimization opportunities | Image compression, minor CLS issues, non-essential schema missing | Next quarter |
+
+### Technical Debt Prioritization Matrix
+
+| Factor | Weight | Assessment |
+|--------|--------|-----------|
+| Pages affected | 30% | Site-wide > Section > Single page |
+| Revenue impact | 25% | Revenue pages > Blog > Utility pages |
+| Fix difficulty | 20% | Config change < Template change < Code rewrite |
+| Competitive impact | 15% | Competitors passing you > parity > you ahead |
+| Crawl budget waste | 10% | High waste > Moderate > Minimal |
+
+## Core Web Vitals Optimization Quick Reference
+
+### LCP (Largest Contentful Paint) Optimization
+
+| Root Cause | Detection | Fix |
+|-----------|-----------|-----|
+| Large hero image | PageSpeed Insights | Serve WebP, resize to container, add loading="lazy" |
+| Render-blocking CSS/JS | DevTools Coverage | Defer non-critical, inline critical CSS |
+| Slow server response | TTFB >800ms | CDN, server-side caching, upgrade hosting |
+| Third-party scripts | DevTools Network | Defer/async, use facade pattern |
+
+### CLS (Cumulative Layout Shift) Optimization
+
+| Root Cause | Detection | Fix |
+|-----------|-----------|-----|
+| Images without dimensions | DevTools | Add explicit width/height attributes |
+| Ads/embeds without reserved space | Visual inspection | Set min-height on containers |
+| Web fonts causing FOUT | DevTools | font-display: swap + preload fonts |
+| Dynamic content injection | Visual inspection | Reserve space with CSS |
+
+### INP (Interaction to Next Paint) Optimization
+
+| Root Cause | Detection | Fix |
+|-----------|-----------|-----|
+| Long JavaScript tasks | DevTools Performance | Break into smaller tasks, use requestIdleCallback |
+| Heavy event handlers | DevTools | Debounce/throttle, use passive listeners |
+| Main thread blocking | DevTools | Web workers for heavy computation |
+