@botlearn/rss-manager 0.1.0

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 BotLearn
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,35 @@
+# @botlearn/rss-manager
+
+> Multi-source RSS/Atom feed aggregation, deduplication, importance scoring, topic clustering, and daily digest generation for OpenClaw Agent
+
+## Installation
+
+```bash
+# via npm
+npm install @botlearn/rss-manager
+
+# via clawhub
+clawhub install @botlearn/rss-manager
+```
+
+## Category
+
+Information Retrieval
+
+## Dependencies
+
+None
+
+## Files
+
+| File | Description |
+|------|-------------|
+| `manifest.json` | Skill metadata and configuration |
+| `skill.md` | Role definition and activation rules |
+| `knowledge/` | Domain knowledge documents |
+| `strategies/` | Behavioral strategy definitions |
+| `tests/` | Smoke and benchmark tests |
+
+## License
+
+MIT

package/knowledge/anti-patterns.md

@@ -0,0 +1,94 @@
+---
+domain: rss-manager
+topic: anti-patterns
+priority: medium
+ttl: 30d
+---
+
+# RSS Manager -- Anti-Patterns
+
+## Feed Parsing Anti-Patterns
+
+### 1. Assuming Well-Formed XML
+- **Problem**: Treating all feeds as valid XML and failing hard on the first parse error. In practice, 5-15% of feeds in the wild contain malformed XML: unescaped ampersands, invalid characters, unclosed tags, or mixed encoding declarations
+- **Fix**: Use a lenient XML parser with fallback strategies. On parse failure: (1) attempt HTML-tolerant parsing, (2) try fixing common issues (unescaped `&`, invalid UTF-8 bytes), (3) attempt regex-based extraction as last resort. Log parse errors per feed for health monitoring
+
+### 2. Ignoring Namespace Prefixes
+- **Problem**: Hardcoding namespace prefixes like `content:encoded` instead of resolving by namespace URI. Feed A might use `content:encoded` while Feed B uses `c:encoded` -- both map to the same namespace URI but a prefix-based parser breaks on Feed B
+- **Fix**: Resolve all elements by their full namespace URI (`http://purl.org/rss/1.0/modules/content/`), not the prefix string. Use a namespace-aware XML parser
+
+### 3. Trusting Feed-Declared Encoding
+- **Problem**: Accepting the XML declaration's encoding attribute (`encoding="UTF-8"`) without verification. Many feeds declare UTF-8 but actually serve Windows-1252 or ISO-8859-1, causing mojibake in non-ASCII characters
+- **Fix**: Detect actual encoding using BOM detection and byte-pattern analysis. If detected encoding conflicts with declared encoding, trust the detection. Always validate that decoded text is valid Unicode before processing
+
+### 4. Ignoring Content Type Negotiation
+- **Problem**: Not sending proper `Accept` headers when requesting feeds, or not checking the response `Content-Type`. Some servers return HTML error pages or redirects with `text/html` instead of the expected feed XML
+- **Fix**: Send `Accept: application/rss+xml, application/atom+xml, application/xml, text/xml;q=0.9` header. Verify response `Content-Type` before parsing. If HTML is returned, attempt feed auto-discovery from the HTML page
+
+## Deduplication Anti-Patterns
+
+### 5. URL-Only Deduplication
+- **Problem**: Deduplicating solely by comparing URLs. This misses: (1) the same article with different tracking parameters, (2) the same article syndicated to multiple domains, (3) updated articles with new URLs, and (4) AMP vs canonical URL variants
+- **Fix**: Use the multi-signal deduplication pipeline from knowledge/best-practices.md. URL matching should be one layer (after canonicalization), not the only layer. Always combine with title similarity and content fingerprinting
+
+### 6. Title-Only Deduplication
+- **Problem**: Deduplicating solely by title match. Short titles like "Q3 Earnings Report" or "Weekly Update" produce massive false positive matches across unrelated feeds. Conversely, slightly reworded titles ("Company X Acquires Y" vs "Y Acquired by Company X") produce false negatives
+- **Fix**: Never use title matching alone. Combine with at least one content-level signal (fingerprint or entity overlap). For titles under 5 words, require additional confirmation signals. Use fuzzy matching with appropriate thresholds per title length
+
+### 7. Aggressive Deduplication Without Clustering
+- **Problem**: Discarding all but one article when duplicates are detected, losing valuable diverse perspectives. If Reuters, BBC, and a domain expert all cover the same event, the domain expert's unique analysis gets discarded
+- **Fix**: Cluster related articles rather than deleting duplicates. Select the most comprehensive article as the "lead" but preserve other sources as "Related" entries. The digest should show source diversity, not suppress it
+
+### 8. Ignoring Article Updates
+- **Problem**: Treating an article with a matching GUID but updated content as a duplicate and ignoring the update. Many feeds legitimately update articles: correcting errors, adding developments, or appending editor notes
+- **Fix**: When a GUID matches but content has changed (detected via content hash), treat it as an article revision. Keep the latest version but note "Updated" in the digest. Track revision history for articles that update frequently
+
+## Importance Scoring Anti-Patterns
+
+### 9. Recency-Only Ranking
+- **Problem**: Ranking articles purely by publication date, treating all newer articles as more important. This surfaces low-quality recent content above high-quality older content and is easily gamed by feeds that backdate or repeatedly update timestamps
+- **Fix**: Use the multi-dimensional scoring model from knowledge/best-practices.md. Recency should be one factor (20% weight), balanced against source authority, cross-source corroboration, topic relevance, and content depth
+
+### 10. Equal Source Weighting
+- **Problem**: Treating all feed sources as equally authoritative. A random blog post and a Reuters wire report about the same event get the same importance score, leading to unreliable content surfacing in top stories
+- **Fix**: Maintain per-source authority tiers (T1-T5) and apply authority weight to importance scoring. Initialize tiers from known source lists and refine based on historical signal quality, factual accuracy, and user feedback
+
+### 11. Ignoring Cross-Source Corroboration
+- **Problem**: Scoring articles independently without considering how many other sources cover the same story. A single blog post about an event gets the same importance as a story covered by 15 major outlets
+- **Fix**: After deduplication clustering, use the cluster size as a corroboration signal. Stories covered by multiple independent sources are more likely to be genuinely important. Weight: cluster_size * source_diversity_factor
+
+### 12. Static Interest Profiles
+- **Problem**: Using a fixed user interest profile that never adapts. The user's interests shift over time, but the digest keeps surfacing topics they no longer care about while missing emerging interests
+- **Fix**: Implement interest profile decay (reduce weight for unengaged topics by 10%/week) and reinforcement (boost topics the user clicks through on). Allow explicit user feedback ("more like this" / "less like this") to directly adjust weights
+
+## Digest Generation Anti-Patterns
+
+### 13. Information Overload Digests
+- **Problem**: Including every article from every feed in the digest, producing an overwhelming wall of text. Users receiving 200+ item digests stop reading them entirely, defeating the purpose of aggregation
+- **Fix**: Apply strict digest sizing limits from knowledge/best-practices.md. Morning briefs: 10-15 top stories with 50-75 word summaries. Use importance scoring to ruthlessly prioritize. Surface detail on demand ("Show me more about this topic") rather than by default
+
+### 14. Flat List Presentation
+- **Problem**: Presenting digest items as a flat chronological list with no organization. Users must scan the entire list to find topics they care about, and related articles about the same event appear scattered throughout
+- **Fix**: Organize digests by topic clusters. Group related articles under topic headings. Within each topic, order by importance score. Provide a table of contents at the top with topic labels and article counts
+
+### 15. Missing Source Attribution
+- **Problem**: Summarizing articles without attributing the original source. Users cannot verify information, assess credibility, or navigate to the full article. Aggregation without attribution also raises ethical and legal concerns
+- **Fix**: Every digest item must include: source name, publication date, direct URL to the original article, and source authority tier. When clustering, list all contributing sources
+
+### 16. Stale Digest Windows
+- **Problem**: Using a fixed 24-hour digest window regardless of user behavior or news cycle. Breaking news gets delayed until the next scheduled digest, while slow news days produce padding with low-quality content
+- **Fix**: Support multiple digest cadences (morning brief, midday update, evening recap). Implement "breaking news" threshold: if an article scores above 90 importance and is from a T1 source, consider immediate notification outside the regular digest schedule
+
+## Feed Health Anti-Patterns
+
+### 17. Silent Feed Failures
+- **Problem**: Continuing to poll feeds that consistently fail (404, 500, parse errors) without alerting the user. The user believes they are monitoring a source that has actually been dead for weeks
+- **Fix**: Track per-feed error rate over a rolling window. After 3 consecutive failures or >50% failure rate over 7 days, mark the feed as unhealthy and alert the user. Suggest alternatives if available (e.g., the feed may have moved to a new URL)
+
+### 18. No Feed Diversity Monitoring
+- **Problem**: Not tracking the topical diversity of subscribed feeds. The user may subscribe to 20 feeds that all cover the same narrow topic, creating an echo chamber with massive duplication and no breadth
+- **Fix**: Periodically analyze the topic distribution across all subscribed feeds. Report: "80% of your feeds cover AI/ML; consider adding feeds for [underrepresented topics based on stated interests]." Show a diversity dashboard with topic coverage breakdown
+
+### 19. Ignoring Feed Freshness Decay
+- **Problem**: Continuing to poll feeds at the same rate even when they haven't published new content in weeks or months. This wastes resources and clutters the feed list with dormant sources
+- **Fix**: Implement adaptive polling (see knowledge/best-practices.md). After 14+ days of no new content, reduce polling to once daily. After 30+ days, reduce to weekly. After 90+ days, prompt the user: "This feed appears inactive. Keep monitoring or unsubscribe?"

package/knowledge/best-practices.md

@@ -0,0 +1,208 @@
+---
+domain: rss-manager
+topic: feed-management-dedup-and-scoring
+priority: high
+ttl: 30d
+---
+
+# RSS Manager -- Best Practices
+
+## Feed Collection & Polling
+
+### 1. Respect Publisher Update Intervals
+
+Before polling a feed, check for update frequency hints:
+- **RSS `<ttl>`** -- Minutes the feed can be cached (e.g., `<ttl>60</ttl>` means poll no more than once per hour)
+- **Atom `<sy:updatePeriod>` and `<sy:updateFrequency>`** -- e.g., `hourly` with frequency `1` means once per hour
+- **HTTP `Cache-Control` / `Expires` headers** -- Standard cache directives
+- **HTTP `ETag` / `Last-Modified` headers** -- Use conditional requests (`If-None-Match`, `If-Modified-Since`) to avoid re-downloading unchanged feeds
+
+Default polling schedule when no hints are available:
+
+| Feed Type | Suggested Interval | Rationale |
+|-----------|-------------------|-----------|
+| Breaking news | 15 minutes | High update frequency expected |
+| Major news outlets | 30 minutes | Regular updates throughout the day |
+| Blog / personal site | 2-4 hours | Updates less frequently |
+| Weekly newsletter | 12-24 hours | Low frequency, conserve resources |
+| Dormant / low-activity | 24 hours | Check daily, reclassify if activity increases |
+
+### 2. Adaptive Polling
+
+Track feed update patterns over time and adjust polling frequency:
+- If a feed hasn't changed in 5 consecutive polls, double the interval (up to 24 hours max)
+- If a feed has new content on every poll, halve the interval (down to the minimum allowed by TTL/headers)
+- Track the average number of new items per poll to predict optimal timing
+- Maintain a per-feed "reliability score" based on: uptime, valid XML rate, consistent timestamps
+
+### 3. Error Handling
+
+- **HTTP 301 (Moved Permanently)** -- Update the stored feed URL
+- **HTTP 410 (Gone)** -- Mark feed as dead; alert the user; stop polling
+- **HTTP 429 (Too Many Requests)** -- Back off using `Retry-After` header; double interval
+- **XML parse failures** -- Attempt recovery with lenient parsing; if persistent (3+ failures), flag feed as unhealthy
+- **Timeout** -- Set a 30-second timeout per feed; retry once after 60 seconds; mark as slow if persistent
+
+## Deduplication Strategies
+
+### Multi-Signal Deduplication Pipeline
+
+Deduplication should use a layered approach, from cheapest to most expensive:
+
+#### Layer 1: GUID / Entry ID Match (Exact)
+- Compare `<guid>` (RSS) or `<id>` (Atom) values directly
+- Cheapest and most reliable when present
+- Caveat: Some feeds reuse GUIDs or change them on updates
+
+#### Layer 2: URL Canonicalization Match (Exact)
+- Canonicalize URLs (see knowledge/domain.md) and compare
+- Catches the same article shared via different tracking URLs
+- Handles `http` vs `https`, `www` vs non-www variants
+
+#### Layer 3: Title Similarity (Fuzzy)
+- Normalize titles: lowercase, strip punctuation, remove common prefixes ("Breaking:", "Update:", "ICYMI:")
+- Use Jaccard similarity on word sets; threshold >= 0.85 indicates a likely duplicate
+- For short titles (< 5 words), require higher threshold (>= 0.95) to avoid false positives
+- Optionally use Levenshtein distance ratio as a secondary signal
+
+#### Layer 4: Content Fingerprinting (Fuzzy)
+- **SimHash**: Compute a 64-bit fingerprint of the article body (after HTML stripping and normalization). Articles with Hamming distance <= 3 are near-duplicates
+- **MinHash with LSH (Locality-Sensitive Hashing)**: Compute k-shingle sets, generate MinHash signatures (128 hashes), use LSH bands (b=16, r=8) for candidate pair detection. Jaccard similarity >= 0.7 confirms near-duplicate
+- **Sentence-level overlap**: Extract the first 3 non-trivial sentences (> 10 words each); if 2+ match another article, flag as near-duplicate
+
+#### Layer 5: Entity & Fact Overlap (Semantic)
+- Extract named entities (people, organizations, locations, dates) from both articles
+- If entity overlap >= 80% and temporal proximity <= 24 hours, likely covering the same event
+- Use this layer to merge "same story, different angle" articles into a cluster rather than discarding
+
+### Dedup Decision Matrix
+
+| Signal Combination | Action |
+|-------------------|--------|
+| GUID match | Exact duplicate -- merge, keep latest version |
+| URL match (after canonicalization) | Exact duplicate -- merge, keep the one with more content |
+| Title similarity >= 0.85 + URL domain differs | Near-duplicate from different sources -- cluster together |
+| Content fingerprint match | Near-duplicate -- cluster together, note source diversity |
+| Entity overlap >= 80% + within 24h | Same event coverage -- cluster, select most comprehensive as primary |
+| Title similarity >= 0.85 + content fingerprint mismatch | Updated/revised article -- keep both, flag as revision |
+
+## Importance Scoring
+
+### Weighted Scoring Model
+
+Score each article on a 0-100 scale using the following weighted dimensions:
+
+| Dimension | Weight | Signal Sources |
+|-----------|--------|---------------|
+| Source Authority | 25% | Domain reputation tier (T1-T5), historical accuracy, feed health score |
+| Recency | 20% | Publication age relative to poll time; decay function: `score = 100 * e^(-0.03 * hours_old)` |
+| Cross-Source Corroboration | 20% | Number of independent sources covering the same story (from dedup clustering) |
+| Topic Relevance | 20% | Cosine similarity between article TF-IDF vector and user interest profile vector |
+| Content Depth | 15% | Word count (normalized), presence of data/citations, structured content (tables, lists) |
+
+### Source Authority Tiers
+
+| Tier | Description | Base Score | Examples |
+|------|-------------|------------|---------|
+| T1 | Wire services, official sources | 90-100 | Reuters, AP, government feeds, RFC publications |
+| T2 | Major established outlets | 75-89 | NYT, BBC, Nature, IEEE Spectrum |
+| T3 | Respected niche/industry sources | 60-74 | TechCrunch, Ars Technica, The Verge, domain-specific journals |
+| T4 | Community & expert blogs | 40-59 | Popular personal blogs, Medium publications with editors, curated newsletters |
+| T5 | Unverified / user-generated | 20-39 | Anonymous blogs, auto-generated feeds, low-quality aggregators |
+
+### Recency Decay Function
+
+Apply an exponential decay to the recency score so that newer articles score higher:
+
+```
+recency_score = 100 * e^(-lambda * hours_since_publication)
+
+lambda = 0.03  (half-life ~ 23 hours)
+```
+
+| Age | Recency Score |
+|-----|--------------|
+| 0 hours | 100 |
+| 6 hours | 84 |
+| 12 hours | 70 |
+| 24 hours | 49 |
+| 48 hours | 24 |
+| 72 hours | 12 |
+
+Adjust lambda per user preference: set lambda lower (e.g., 0.01) for users who prefer weekly digests; higher (e.g., 0.05) for real-time monitoring.
+
+### User Interest Profile
+
+Maintain a vector of topic weights representing the user's interests:
+- Initialize from explicit topic subscriptions and feed categories
+- Update dynamically based on click-through behavior (articles the user reads vs skips)
+- Decay stale interests: reduce weight by 10% per week for topics the user hasn't engaged with
+- Use the interest profile vector to compute topic relevance scores via cosine similarity with article TF-IDF vectors
+
+## Topic Clustering
+
+### TF-IDF Vectorization
+
+1. Preprocess text: lowercase, remove stop words, apply stemming or lemmatization
+2. Compute TF-IDF vectors using the corpus of articles from the current digest window
+3. Use only the title + first 200 words of the body for efficiency
+4. Maintain a background IDF model updated weekly from all ingested articles
+
+### Clustering Algorithm Selection
+
+| Method | When to Use | Parameters |
+|--------|------------|------------|
+| DBSCAN | Default choice; handles variable cluster sizes and noise well | eps=0.4 (cosine distance), min_samples=2 |
+| Agglomerative (Ward) | When you need a fixed number of topics (e.g., "give me 5 topics") | n_clusters=k, linkage=ward |
+| Online K-Means | Streaming / real-time updates where articles arrive continuously | n_clusters=k (estimated from historical data) |
+
+### Cluster Labeling
+
+For each cluster, generate a human-readable topic label:
+1. Extract the top 3 TF-IDF terms from the cluster centroid
+2. Identify the most common named entity (person, org, or location) across cluster articles
+3. Combine into a label: "[Named Entity]: [Top Terms]" (e.g., "OpenAI: language model GPT release")
+4. If no clear named entity, use the top 3 terms as the label
+
+### Representative Article Selection
+
+From each cluster, select one "lead" article to represent the topic:
+1. Pick the article closest to the cluster centroid (most representative)
+2. Break ties by importance score (higher is better)
+3. Break further ties by source authority tier (prefer T1/T2)
+4. Include the lead article's summary in the digest; list other cluster articles as "Related"
+
+## Digest Generation
+
+### Digest Structure
+
+```
+# Daily Digest -- [Date]
+## Top Stories (importance >= 70)
+  [Topic Label 1]
+    - Lead article summary (source, date, importance score)
+    - Related: [n] more articles from [sources]
+  [Topic Label 2]
+    - ...
+
+## Noteworthy (importance 40-69)
+  [Topic clusters organized by category]
+
+## Also Mentioned (importance < 40)
+  [Brief one-line entries]
+
+## Feed Health Report
+  - [n] feeds polled, [n] successful, [n] errors
+  - [n] new articles, [n] duplicates removed
+  - Emerging topic: [topic gaining traction]
+  - Declining topic: [topic losing traction]
+```
+
+### Digest Sizing
+
+| Digest Type | Max Stories | Max Words per Summary | Delivery Window |
+|-------------|-----------|---------------------|----------------|
+| Morning brief | 10-15 | 50-75 | 06:00-08:00 local |
+| Midday update | 5-10 | 30-50 | 11:30-13:00 local |
+| Evening recap | 10-15 | 50-75 | 17:00-19:00 local |
+| Weekly roundup | 20-30 | 100-150 | Saturday/Sunday morning |

package/knowledge/domain.md

@@ -0,0 +1,203 @@
+---
+domain: rss-manager
+topic: feed-formats-parsing-and-content-extraction
+priority: high
+ttl: 30d
+---
+
+# RSS/Atom Feed Formats, XML Parsing & Content Extraction
+
+## Feed Format Specifications
+
+### RSS 2.0 (Really Simple Syndication)
+
+RSS 2.0 is the most widely deployed syndication format. A valid RSS 2.0 document is XML with a root `<rss>` element containing a single `<channel>`.
+
+#### Channel-Level Elements
+
+| Element | Required | Description |
+|---------|----------|-------------|
+| `<title>` | Yes | Name of the feed (e.g., "TechCrunch") |
+| `<link>` | Yes | URL of the HTML website associated with the feed |
+| `<description>` | Yes | Summary of what the feed contains |
+| `<language>` | No | Language code (e.g., "en-us", "zh-cn") |
+| `<lastBuildDate>` | No | Last time feed content changed (RFC 822 date) |
+| `<pubDate>` | No | Publication date of the feed content (RFC 822 date) |
+| `<ttl>` | No | Minutes the feed can be cached before refresh |
+| `<image>` | No | Channel logo with `<url>`, `<title>`, `<link>` sub-elements |
+| `<generator>` | No | Software that generated the feed |
+| `<managingEditor>` | No | Email of the editorial contact |
+| `<category>` | No | One or more categories for the feed |
+
+#### Item-Level Elements
+
+| Element | Required | Description |
+|---------|----------|-------------|
+| `<title>` | Conditional | Title of the article (required if no description) |
+| `<link>` | No | URL of the full article |
+| `<description>` | Conditional | Article summary or full content (required if no title) |
+| `<author>` | No | Email address of the author |
+| `<category>` | No | One or more categories |
+| `<pubDate>` | No | Publication date (RFC 822 format) |
+| `<guid>` | No | Globally unique identifier; `isPermaLink="true"` means it is a URL |
+| `<enclosure>` | No | Attached media; attributes: `url`, `length`, `type` |
+| `<comments>` | No | URL of the comments page |
+| `<source>` | No | Original feed the item came from; attribute: `url` |
+
+#### RSS 2.0 Namespace Extensions
+
+Common namespace extensions enrich standard RSS:
+
+- **`content:encoded`** (xmlns:content="http://purl.org/rss/1.0/modules/content/") — Full HTML content body, preferred over `<description>` for complete article text
+- **`dc:creator`** (xmlns:dc="http://purl.org/dc/elements/1.1/") — Dublin Core author name (more reliable than `<author>`)
+- **`dc:date`** — ISO 8601 date (more precise than `<pubDate>`)
+- **`slash:comments`** — Comment count (integer)
+- **`wfw:commentRss`** — RSS feed URL for the item's comments
+- **`media:content`** — Rich media attachments with `url`, `medium`, `type`, `width`, `height`
+- **`media:thumbnail`** — Thumbnail image URL
+
+### RSS 1.0 (RDF Site Summary)
+
+RSS 1.0 is RDF-based and uses XML namespaces extensively. Less common than RSS 2.0 but still found in academic and government feeds.
+
+#### Key Differences from RSS 2.0
+
+- Root element is `<rdf:RDF>` (not `<rss>`)
+- Items are listed both inside `<channel><items><rdf:Seq>` as references and as top-level `<item>` elements
+- Uses `rdf:about` attribute for resource identification
+- Relies heavily on Dublin Core (`dc:`) namespace for metadata
+- Extensible through RDF modules: `mod_syndication` (update schedule), `mod_taxonomy` (topic classification)
+
+### Atom 1.0 (RFC 4287)
+
+Atom is a more formally specified format than RSS, with clearer semantics and mandatory fields.
+
+#### Feed-Level Elements
+
+| Element | Required | Description |
+|---------|----------|-------------|
+| `<title>` | Yes | Feed title (supports `type` attribute: text, html, xhtml) |
+| `<id>` | Yes | Permanent, universally unique feed identifier (IRI) |
+| `<updated>` | Yes | Last time the feed was modified (RFC 3339 / ISO 8601) |
+| `<author>` | Yes | At least one `<author>` with `<name>`, optional `<email>`, `<uri>` |
+| `<link>` | Yes | Must include `rel="self"` (feed URL) and `rel="alternate"` (website URL) |
+| `<subtitle>` | No | Feed description |
+| `<generator>` | No | Software that generated the feed |
+| `<icon>` | No | Small feed icon URL |
+| `<logo>` | No | Feed logo URL |
+| `<rights>` | No | Copyright notice |
+| `<category>` | No | One or more categories with `term`, `scheme`, `label` attributes |
+
+#### Entry-Level Elements
+
+| Element | Required | Description |
+|---------|----------|-------------|
+| `<title>` | Yes | Entry title (with `type` attribute) |
+| `<id>` | Yes | Permanent unique identifier for the entry (IRI) |
+| `<updated>` | Yes | Last modification timestamp |
+| `<published>` | No | Original publication timestamp |
+| `<author>` | Conditional | Required if feed-level author is absent |
+| `<content>` | Recommended | Full entry content; `type` attribute: text, html, xhtml, or media type |
+| `<summary>` | Recommended | Short summary; required if `<content>` is absent or non-text |
+| `<link>` | Recommended | `rel="alternate"` for the article URL |
+| `<category>` | No | One or more categories |
+| `<contributor>` | No | Additional contributors |
+| `<source>` | No | Original feed metadata if the entry was aggregated |
+
+## XML Parsing Considerations
+
+### Encoding Detection Priority
+
+1. HTTP `Content-Type` header charset (highest priority)
+2. XML declaration encoding attribute: `<?xml version="1.0" encoding="UTF-8"?>`
+3. BOM (Byte Order Mark) detection
+4. Default to UTF-8 if none specified
+
+### Common Encoding Issues
+
+- **Double encoding**: Content encoded as UTF-8 then re-encoded, producing mojibake (e.g., `é` instead of `e`)
+- **Windows-1252 mislabeled as ISO-8859-1**: Characters in the 0x80-0x9F range render incorrectly
+- **HTML entities in XML**: `&nbsp;` is valid in HTML but not in XML -- must use `&#160;` or be wrapped in CDATA
+- **Unescaped ampersands**: `&` in URLs or text breaks XML parsing -- must be `&amp;`
+
+### CDATA Section Handling
+
+Many feeds wrap HTML content in CDATA sections to avoid XML escaping issues:
+
+```xml
+<description><![CDATA[<p>Article with <a href="https://example.com">links</a> and <img src="photo.jpg" /></p>]]></description>
+```
+
+Processing steps:
+1. Extract raw content from CDATA (stripping `<![CDATA[` and `]]>`)
+2. Parse the inner HTML separately
+3. Sanitize: strip `<script>`, `<iframe>`, event handlers, `javascript:` URIs
+4. Extract plain text for indexing; preserve HTML for display
+
+### Namespace Resolution
+
+Feeds commonly use multiple namespaces. A robust parser must:
+
+1. Resolve namespace prefixes to their URIs (prefixes may differ between feeds)
+2. Recognize elements by namespace URI, not prefix (e.g., `content:encoded` and `c:encoded` are the same if both map to `http://purl.org/rss/1.0/modules/content/`)
+3. Handle default namespace declarations on the root element
+4. Gracefully ignore unknown namespaces rather than failing
+
+## Content Extraction
+
+### Extracting Article Text
+
+Priority order for obtaining article body:
+
+1. **Atom `<content type="html">`** or **`<content type="xhtml">`** -- fullest content
+2. **RSS `<content:encoded>`** -- full HTML body (namespace extension)
+3. **Atom `<summary>`** -- may be full text or truncated
+4. **RSS `<description>`** -- may be full text, truncated, or just a snippet
+5. **Fetch the linked URL** -- fallback when feed only provides a title or minimal snippet
+
+### Metadata Extraction Checklist
+
+For each feed item, extract and normalize:
+
+| Field | Primary Source | Fallback | Normalization |
+|-------|---------------|----------|---------------|
+| Title | `<title>` | First line of description | Strip HTML, decode entities, trim whitespace |
+| URL | `<link>` / `<guid isPermaLink="true">` | `<id>` (Atom) | Canonicalize: lowercase host, remove tracking params |
+| Author | `<dc:creator>` / `<author><name>` | `<author>` (email) / `<managingEditor>` | Extract name, discard email if present |
+| Date | `<dc:date>` / `<published>` / `<updated>` | `<pubDate>` | Parse to ISO 8601 UTC; handle RFC 822, RFC 3339, and common non-standard formats |
+| Body | `<content:encoded>` / `<content>` | `<description>` / `<summary>` | Sanitize HTML, extract plain text, calculate word count |
+| Categories | `<category>` (multiple) | `<dc:subject>` | Normalize casing, map synonyms |
+| Media | `<enclosure>` / `<media:content>` | `<media:thumbnail>` / embedded `<img>` | Extract URL, MIME type, dimensions |
+| GUID | `<guid>` / `<id>` | URL | Use as-is for deduplication key |
+
+### Date Parsing
+
+Feeds use inconsistent date formats. A robust parser must handle:
+
+- **RFC 822**: `Mon, 15 Jan 2024 13:45:00 GMT` (RSS 2.0 standard)
+- **RFC 3339 / ISO 8601**: `2024-01-15T13:45:00Z` (Atom standard)
+- **Non-standard variations**: `Jan 15, 2024`, `2024/01/15`, `15-01-2024`, `1705322700` (Unix timestamp)
+- **Timezone ambiguity**: `EST` vs `-0500`; always convert to UTC for consistent comparison
+- **Missing timezone**: Assume UTC and flag as uncertain
+
+### URL Canonicalization
+
+To detect duplicate URLs pointing to the same article:
+
+1. Convert scheme and host to lowercase: `HTTPS://WWW.Example.COM` -> `https://www.example.com`
+2. Remove default ports: `:80` for HTTP, `:443` for HTTPS
+3. Remove trailing slashes on paths (unless the path is `/`)
+4. Sort query parameters alphabetically
+5. Remove known tracking parameters: `utm_source`, `utm_medium`, `utm_campaign`, `utm_content`, `utm_term`, `ref`, `source`, `fbclid`, `gclid`, `mc_cid`, `mc_eid`
+6. Decode unnecessary percent-encoding: `%41` -> `A`
+7. Remove fragment identifiers (`#section`) unless they are part of a single-page app route
+
+### Feed Discovery
+
+When given a website URL instead of a feed URL, discover feeds by:
+
+1. Check `<link rel="alternate" type="application/rss+xml">` in HTML `<head>`
+2. Check `<link rel="alternate" type="application/atom+xml">` in HTML `<head>`
+3. Try common paths: `/feed`, `/rss`, `/atom.xml`, `/feed.xml`, `/rss.xml`, `/index.xml`, `/feeds/posts/default` (Blogger)
+4. Check `/.well-known/` resources
+5. Parse the page for embedded feed links in the body content

package/manifest.json

@@ -0,0 +1,26 @@
+{
+  "name": "@botlearn/rss-manager",
+  "version": "0.1.0",
+  "description": "Multi-source RSS/Atom feed aggregation, deduplication, importance scoring, topic clustering, and daily digest generation for OpenClaw Agent",
+  "category": "information-retrieval",
+  "author": "BotLearn",
+  "benchmarkDimension": "information-retrieval",
+  "expectedImprovement": 30,
+  "dependencies": {},
+  "compatibility": {
+    "openclaw": ">=0.5.0"
+  },
+  "files": {
+    "skill": "skill.md",
+    "knowledge": [
+      "knowledge/domain.md",
+      "knowledge/best-practices.md",
+      "knowledge/anti-patterns.md"
+    ],
+    "strategies": [
+      "strategies/main.md"
+    ],
+    "smokeTest": "tests/smoke.json",
+    "benchmark": "tests/benchmark.json"
+  }
+}

package/package.json

@@ -0,0 +1,35 @@
+{
+  "name": "@botlearn/rss-manager",
+  "version": "0.1.0",
+  "description": "Multi-source RSS/Atom feed aggregation, deduplication, importance scoring, topic clustering, and daily digest generation for OpenClaw Agent",
+  "type": "module",
+  "main": "manifest.json",
+  "files": [
+    "manifest.json",
+    "skill.md",
+    "knowledge/",
+    "strategies/",
+    "tests/",
+    "README.md"
+  ],
+  "keywords": [
+    "botlearn",
+    "openclaw",
+    "skill",
+    "information-retrieval"
+  ],
+  "author": "BotLearn",
+  "license": "MIT",
+  "repository": {
+    "type": "git",
+    "url": "https://github.com/readai-team/botlearn-awesome-skills.git",
+    "directory": "packages/skills/rss-manager"
+  },
+  "homepage": "https://github.com/readai-team/botlearn-awesome-skills/tree/main/packages/skills/rss-manager",
+  "bugs": {
+    "url": "https://github.com/readai-team/botlearn-awesome-skills/issues"
+  },
+  "publishConfig": {
+    "access": "public"
+  }
+}