@rudderhq/server 0.1.0-canary.0

package/resources/community-skills/deep-research/reference/report-assembly.md

@@ -0,0 +1,119 @@
+# Report Assembly: Progressive File Generation
+
+## Length Requirements by Mode
+
+| Mode | Target Words | Description |
+|------|--------------|-------------|
+| Quick | 2,000-4,000 | Baseline quality threshold |
+| Standard | 4,000-8,000 | Comprehensive analysis |
+| Deep | 8,000-15,000 | Thorough investigation |
+| UltraDeep | 15,000-20,000+ | Maximum rigor (at output limit) |
+
+---
+
+## Output Token Safeguard
+
+**Claude Code default limit:** 32,000 output tokens (~24,000 words total per execution)
+
+**Practical limits:**
+- Target <=20,000 words total output
+- Leave safety margin for tool call overhead
+- Reports >20,000 words require auto-continuation (see continuation.md)
+
+---
+
+## Progressive Section Generation
+
+**Core Strategy:** Generate and write each section individually using Write/Edit tools. This allows unlimited report length while keeping each generation manageable.
+
+### Phase 8.1: Setup
+
+```bash
+# Create folder: ~/Documents/[TopicName]_Research_[YYYYMMDD]/
+mkdir -p ~/Documents/[folder_name]
+
+# Initialize markdown file with frontmatter
+# Path: [folder]/research_report_[YYYYMMDD]_[slug].md
+```
+
+### Phase 8.2: Section Generation Loop
+
+**Pattern:** Generate section -> Write/Edit to file -> Move to next section
+Each Write/Edit call contains ONE section (<=2,000 words per call)
+
+**Initialize citation tracking (persist to disk):**
+```bash
+# Create sources.json in the report folder for durable provenance
+# Each entry: {"num": N, "title": "...", "url": "...", "claim": "...", "evidence_quote": "..."}
+echo '[]' > [folder]/sources.json
+```
+Update sources.json after each section. This survives context compaction and enables continuation agents to pick up citation state.
+
+**Section sequence:**
+
+1. **Executive Summary** (200-400 words)
+   - Tool: Write(file, frontmatter + Executive Summary)
+   - Track citations
+   - Progress: "Executive Summary complete"
+
+2. **Introduction** (400-800 words)
+   - Tool: Edit(file, append Introduction)
+   - Track citations
+   - Progress: "Introduction complete"
+
+3. **Finding 1-N** (600-2,000 words each)
+   - Tool: Edit(file, append Finding N)
+   - Track citations
+   - Include at least one concrete example, mini-case, counterexample, or applied scenario when it improves comprehension
+   - If the example is hypothetical, label it clearly rather than citing it as sourced fact
+   - Progress: "Finding N complete"
+
+4. **Synthesis & Insights**
+   - Novel insights beyond source statements
+   - Add a concrete scenario or mini-case if synthesis is otherwise too abstract
+   - Tool: Edit(append)
+
+5. **Limitations & Caveats**
+   - Counterevidence, gaps, uncertainties
+   - Tool: Edit(append)
+
+6. **Recommendations**
+   - Immediate actions, next steps, research needs
+   - Where useful, show how one recommendation would play out in a concrete applied case
+   - Tool: Edit(append)
+
+7. **Bibliography** (CRITICAL)
+   - EVERY citation from citations_used list
+   - NO ranges, NO placeholders, NO truncation
+   - Tool: Edit(append)
+
+8. **Methodology Appendix**
+   - Research process, verification approach
+   - Tool: Edit(append)
+
+---
+
+## File Organization
+
+**1. Create dedicated folder:**
+- Location: `~/Documents/[TopicName]_Research_[YYYYMMDD]/`
+- Clean topic name (remove special chars, use underscores)
+
+**2. File naming convention:**
+All files use same base name:
+- `research_report_20251104_topic_slug.md`
+- `research_report_20251104_topic_slug.html`
+- `research_report_20251104_topic_slug.pdf`
+
+**3. Also save copy to:** `~/.claude/research_output/` (internal tracking)
+
+---
+
+## Word Count Per Section
+
+**CRITICAL:** No single Edit call should exceed 2,000 words.
+
+Example: 10 findings x 1,500 words = 15,000 words total
+- Each Edit call: 1,500 words (under limit)
+- File grows to 15,000 words
+- No single tool call exceeds limits

package/resources/community-skills/deep-research/reference/weasyprint_guidelines.md

@@ -0,0 +1,324 @@
+# WeasyPrint PDF Generation Guidelines
+
+## Overview
+
+WeasyPrint converts HTML/CSS to PDF. These guidelines ensure professional output without awkward page breaks, orphaned content, or layout issues.
+
+---
+
+## Critical CSS Properties for Page Breaks
+
+### Prevent Breaking Inside Elements
+
+```css
+/* Apply to containers that should never split across pages */
+.executive-summary,
+.key-insight,
+.warning-box,
+.action-box,
+.diagram,
+.metrics-row,
+table {
+    page-break-inside: avoid;
+}
+
+/* Tables are especially problematic - always prevent breaks */
+table {
+    page-break-inside: avoid;
+}
+
+/* Two-column layouts */
+.two-col {
+    page-break-inside: avoid;
+}
+```
+
+### Prevent Orphaned Headers
+
+```css
+/* Headers should never appear at bottom of page without content */
+h2, h3, h4 {
+    page-break-after: avoid;
+}
+```
+
+### Prevent Widows and Orphans in Text
+
+```css
+p {
+    orphans: 3;  /* Minimum lines at bottom of page */
+    widows: 3;   /* Minimum lines at top of page */
+}
+```
+
+---
+
+## @page Rules
+
+### Basic Setup
+
+```css
+@page {
+    size: A4;
+    margin: 25mm 20mm 25mm 20mm;
+
+    @top-center {
+        content: "Report Title";
+        font-family: Georgia, serif;
+        font-size: 9pt;
+        color: #666666;
+    }
+
+    @bottom-center {
+        content: counter(page);
+        font-family: Georgia, serif;
+        font-size: 10pt;
+    }
+}
+
+/* Suppress header on first page */
+@page :first {
+    @top-center { content: none; }
+}
+```
+
+---
+
+## Table Design for PDF
+
+### Avoid Large Tables
+
+- Keep tables under 8-10 rows when possible
+- Split large data sets into multiple smaller tables
+- Use `page-break-inside: avoid` on every table
+
+### Table CSS
+
+```css
+table {
+    width: 100%;
+    border-collapse: collapse;
+    margin: 12pt 0;
+    font-size: 9pt;
+    page-break-inside: avoid;
+}
+
+th {
+    background: #1a1a1a;
+    color: white;
+    padding: 8pt 10pt;
+    text-align: left;
+    font-size: 8pt;
+    text-transform: uppercase;
+}
+
+td {
+    padding: 8pt 10pt;
+    border-bottom: 0.5pt solid #d0d0d0;
+    vertical-align: top;
+}
+```
+
+---
+
+## Typography for Print
+
+### Font Sizes (pt not px)
+
+Use points for print, not pixels:
+
+```css
+body {
+    font-family: Georgia, "Times New Roman", Times, serif;
+    font-size: 10pt;
+    line-height: 1.6;
+}
+
+h1 { font-size: 22pt; }
+h2 { font-size: 14pt; }
+h3 { font-size: 11pt; }
+
+/* Small text */
+.citation { font-size: 8pt; }
+.footer { font-size: 8pt; }
+.bib-entry { font-size: 8pt; }
+```
+
+### Line Height
+
+- Body text: 1.6-1.7
+- Tables: 1.4-1.5
+- Bibliography: 1.5
+
+---
+
+## Layout Patterns That Work
+
+### Use `display: table` for Side-by-Side
+
+Flexbox and Grid have limited WeasyPrint support. Use `display: table`:
+
+```css
+.two-col {
+    display: table;
+    width: 100%;
+    page-break-inside: avoid;
+}
+
+.col {
+    display: table-cell;
+    width: 50%;
+    padding: 10pt;
+    vertical-align: top;
+}
+
+.col:first-child {
+    border-right: 0.5pt solid #cccccc;
+}
+```
+
+### Metrics Dashboard
+
+```css
+.metrics-row {
+    display: table;
+    width: 100%;
+    border: 1.5pt solid #000000;
+    page-break-inside: avoid;
+}
+
+.metric {
+    display: table-cell;
+    width: 25%;
+    padding: 12pt 8pt;
+    text-align: center;
+}
+```
+
+---
+
+## Content Boxes
+
+### Insight/Warning Boxes
+
+```css
+.key-insight {
+    background: #f5f5f5;
+    border-left: 3pt solid #000000;
+    padding: 10pt 12pt;
+    margin: 12pt 0;
+    page-break-inside: avoid;
+}
+
+.warning-box {
+    background: #1a1a1a;
+    color: white;
+    padding: 12pt 15pt;
+    margin: 12pt 0;
+    page-break-inside: avoid;
+}
+```
+
+### Diagrams
+
+```css
+.diagram {
+    background: #f5f5f5;
+    border: 1pt solid #000000;
+    padding: 12pt;
+    margin: 12pt 0;
+    text-align: center;
+    page-break-inside: avoid;
+}
+```
+
+---
+
+## Bibliography
+
+```css
+.bibliography {
+    background: #f5f5f5;
+    padding: 15pt;
+    margin-top: 20pt;
+    border-top: 2pt solid #000000;
+}
+
+.bib-entry {
+    margin-bottom: 8pt;
+    padding-left: 25pt;
+    text-indent: -25pt;
+    font-size: 8pt;
+    line-height: 1.5;
+    page-break-inside: avoid;
+}
+```
+
+---
+
+## Common Problems and Solutions
+
+### Problem: Table Splits Across Pages
+
+**Solution:** Add `page-break-inside: avoid` to table. If table is too large, split into multiple smaller tables.
+
+### Problem: Header at Bottom of Page with No Content
+
+**Solution:** Add `page-break-after: avoid` to all heading elements.
+
+### Problem: Single Line at Top/Bottom of Page
+
+**Solution:** Set `orphans: 3` and `widows: 3` on paragraphs.
+
+### Problem: Flex/Grid Layout Breaks
+
+**Solution:** Use `display: table` and `display: table-cell` instead.
+
+### Problem: Images/Diagrams Cut Off
+
+**Solution:** Add `page-break-inside: avoid` to container.
+
+### Problem: Margins Too Tight
+
+**Solution:** Use generous @page margins (25mm top/bottom, 20mm sides).
+
+---
+
+## Compact Report Strategy
+
+To reduce page count while maintaining readability:
+
+1. **Use 10pt base font** (not 12pt)
+2. **Tighter line-height**: 1.5-1.6 instead of 1.8
+3. **Smaller margins in boxes**: 10pt padding instead of 15pt
+4. **Condensed bibliography**: 8pt font, tighter spacing
+5. **Two-column layouts** for comparison data
+6. **Inline metrics dashboard** rather than full-width cards
+
+---
+
+## Validation Checklist
+
+Before generating PDF, verify:
+
+- [ ] All tables have `page-break-inside: avoid`
+- [ ] All boxed content has `page-break-inside: avoid`
+- [ ] Headers have `page-break-after: avoid`
+- [ ] Paragraphs have `orphans: 3; widows: 3`
+- [ ] No Flexbox or Grid in critical layouts
+- [ ] Font sizes in pt, not px
+- [ ] @page margins defined
+- [ ] Two-column layouts use `display: table`
+
+---
+
+## Generation Command
+
+```bash
+weasyprint input.html output.pdf
+```
+
+Options:
+- `--presentational-hints` - Respect HTML presentational hints
+- `-s stylesheet.css` - Apply external stylesheet
+- `--pdf-variant pdf/ua-1` - Generate accessible PDF

package/resources/community-skills/deep-research/requirements.txt

@@ -0,0 +1,14 @@
+# Deep Research Skill Dependencies
+#
+# Core: Python 3.9+ standard library only. No pip install needed.
+#
+# Optional tools (not Python packages):
+#
+# search-cli — multi-provider search aggregation (Brave, Serper, Exa, Jina, Firecrawl)
+#   Install: brew tap 199-biotechnologies/tap && brew install search-cli
+#   Config:  search config set keys.[provider] YOUR_KEY
+#   Repo:    https://github.com/199-biotechnologies/search-cli
+#
+# weasyprint — PDF generation from HTML reports
+#   Install: pip install weasyprint
+#   Used by: reference/html-generation.md (Phase 8 PDF output)

package/resources/community-skills/deep-research/scripts/citation_manager.py

@@ -0,0 +1,177 @@
+#!/usr/bin/env python3
+"""
+Citation Management System
+Tracks sources, generates citations, and maintains bibliography
+"""
+
+from dataclasses import dataclass, field
+from typing import List, Dict, Optional
+from datetime import datetime
+from urllib.parse import urlparse
+import hashlib
+
+
+@dataclass
+class Citation:
+    """Represents a single citation"""
+    id: str
+    title: str
+    url: str
+    authors: Optional[List[str]] = None
+    publication_date: Optional[str] = None
+    retrieved_date: str = field(default_factory=lambda: datetime.now().strftime('%Y-%m-%d'))
+    source_type: str = "web"  # web, academic, documentation, book, paper
+    doi: Optional[str] = None
+    citation_count: int = 0
+
+    def to_apa(self, index: int) -> str:
+        """Generate APA format citation"""
+        author_str = ""
+        if self.authors:
+            if len(self.authors) == 1:
+                author_str = f"{self.authors[0]}."
+            elif len(self.authors) == 2:
+                author_str = f"{self.authors[0]} & {self.authors[1]}."
+            else:
+                author_str = f"{self.authors[0]} et al."
+
+        date_str = f"({self.publication_date})" if self.publication_date else "(n.d.)"
+
+        return f"[{index}] {author_str} {date_str}. {self.title}. Retrieved {self.retrieved_date}, from {self.url}"
+
+    def to_inline(self, index: int) -> str:
+        """Generate inline citation [index]"""
+        return f"[{index}]"
+
+    def to_markdown(self, index: int) -> str:
+        """Generate markdown link format"""
+        return f"[{index}] [{self.title}]({self.url}) (Retrieved: {self.retrieved_date})"
+
+
+class CitationManager:
+    """Manages citations and bibliography"""
+
+    def __init__(self):
+        self.citations: Dict[str, Citation] = {}
+        self.citation_order: List[str] = []
+
+    def add_source(
+        self,
+        url: str,
+        title: str,
+        authors: Optional[List[str]] = None,
+        publication_date: Optional[str] = None,
+        source_type: str = "web",
+        doi: Optional[str] = None
+    ) -> str:
+        """Add a source and return its citation ID"""
+        # Generate unique ID based on URL
+        citation_id = hashlib.md5(url.encode()).hexdigest()[:8]
+
+        if citation_id not in self.citations:
+            citation = Citation(
+                id=citation_id,
+                title=title,
+                url=url,
+                authors=authors,
+                publication_date=publication_date,
+                source_type=source_type,
+                doi=doi
+            )
+            self.citations[citation_id] = citation
+            self.citation_order.append(citation_id)
+
+        # Increment citation count
+        self.citations[citation_id].citation_count += 1
+
+        return citation_id
+
+    def get_citation_number(self, citation_id: str) -> Optional[int]:
+        """Get the citation number for a given ID"""
+        try:
+            return self.citation_order.index(citation_id) + 1
+        except ValueError:
+            return None
+
+    def get_inline_citation(self, citation_id: str) -> str:
+        """Get inline citation marker [n]"""
+        num = self.get_citation_number(citation_id)
+        return f"[{num}]" if num else "[?]"
+
+    def generate_bibliography(self, style: str = "markdown") -> str:
+        """Generate full bibliography"""
+        if style == "markdown":
+            lines = ["## Bibliography\n"]
+            for i, citation_id in enumerate(self.citation_order, 1):
+                citation = self.citations[citation_id]
+                lines.append(citation.to_markdown(i))
+            return "\n".join(lines)
+
+        elif style == "apa":
+            lines = ["## Bibliography\n"]
+            for i, citation_id in enumerate(self.citation_order, 1):
+                citation = self.citations[citation_id]
+                lines.append(citation.to_apa(i))
+            return "\n".join(lines)
+
+        return "Unsupported citation style"
+
+    def get_statistics(self) -> Dict[str, any]:
+        """Get citation statistics"""
+        return {
+            'total_sources': len(self.citations),
+            'total_citations': sum(c.citation_count for c in self.citations.values()),
+            'source_types': self._count_by_type(),
+            'most_cited': self._get_most_cited(5),
+            'uncited': self._get_uncited()
+        }
+
+    def _count_by_type(self) -> Dict[str, int]:
+        """Count sources by type"""
+        counts = {}
+        for citation in self.citations.values():
+            counts[citation.source_type] = counts.get(citation.source_type, 0) + 1
+        return counts
+
+    def _get_most_cited(self, n: int = 5) -> List[tuple]:
+        """Get most cited sources"""
+        sorted_citations = sorted(
+            self.citations.items(),
+            key=lambda x: x[1].citation_count,
+            reverse=True
+        )
+        return [(self.get_citation_number(cid), c.title, c.citation_count)
+                for cid, c in sorted_citations[:n]]
+
+    def _get_uncited(self) -> List[str]:
+        """Get sources that were added but never cited"""
+        return [c.title for c in self.citations.values() if c.citation_count == 0]
+
+    def export_to_file(self, filepath: str, style: str = "markdown"):
+        """Export bibliography to file"""
+        with open(filepath, 'w') as f:
+            f.write(self.generate_bibliography(style))
+
+
+# Example usage
+if __name__ == '__main__':
+    manager = CitationManager()
+
+    # Add sources
+    id1 = manager.add_source(
+        url="https://example.com/article1",
+        title="Understanding Deep Research",
+        authors=["Smith, J.", "Johnson, K."],
+        publication_date="2025"
+    )
+
+    id2 = manager.add_source(
+        url="https://example.com/article2",
+        title="AI Research Methods",
+        source_type="academic"
+    )
+
+    # Use citations
+    print(f"Inline citation: {manager.get_inline_citation(id1)}")
+    print(f"\nBibliography:\n{manager.generate_bibliography()}")
+    print(f"\nStatistics:\n{manager.get_statistics()}")