@mclawnet/agent 0.5.9 → 0.6.2

package/skills/data-analysis/SKILL.md

@@ -0,0 +1,194 @@
+---
+name: data-analysis
+description: Guide systematic data analysis workflows using Python (pandas, DuckDB) or SQL. Use when analyzing datasets, generating statistics, creating summaries, or exploring structured data from CSV/Excel/database sources.
+---
+
+# Data Analysis
+
+A systematic framework for analyzing structured data — from initial inspection through statistical analysis to visualization and reporting.
+
+## Overview
+
+Data analysis follows a predictable arc:
+
+1. **Understand** — What question are we answering? What decision does this inform?
+2. **Inspect** — What does the data look like? Types, ranges, quality issues?
+3. **Transform** — Clean, reshape, and enrich the data for analysis.
+4. **Analyze** — Apply aggregation or statistical techniques to extract insights.
+5. **Visualize** — Create charts that communicate findings clearly.
+6. **Report** — Summarize findings with context, caveats, and recommendations.
+
+Never skip the inspection step. Jumping straight to analysis on data you do not understand produces confidently wrong results.
+
+## When to Use
+
+- Exploring a dataset's structure and contents
+- Summary statistics, distributions, or trend analysis
+- Answering business questions from CSV, Excel, or database data
+- Aggregation reports with grouping, filtering, and ranking
+- Comparing cohorts or time periods; detecting anomalies
+
+## When NOT to Use
+
+- **ML model training** — This covers descriptive analysis, not predictive modeling.
+- **Real-time streaming** — Use stream processing tools.
+- **ETL pipeline design** — This is ad-hoc analysis, not production pipelines.
+
+## Step 1: Understand Requirements
+
+Before touching data, clarify the question.
+
+- What specific question needs an answer? Restate it precisely.
+- Who is the audience? Technical team, executives, external stakeholders?
+- What format is expected? Table, chart, single number, written summary?
+- What time range, filters, or segments apply?
+
+A vague question like "analyze our sales data" must be narrowed: "Top 10 products by revenue in Q1 2025, broken down by region?"
+
+## Step 2: Data Inspection
+
+**Pandas:**
+```python
+df = pd.read_csv("data.csv")
+df.shape; df.dtypes; df.head(10); df.describe()
+df.isnull().sum(); df.nunique(); df.duplicated().sum()
+```
+
+**DuckDB:**
+```sql
+DESCRIBE TABLE 'data.csv';
+SELECT COUNT(*) FROM 'data.csv';
+SUMMARIZE SELECT * FROM 'data.csv';
+```
+
+**What to note:** Column types match expectations? Missing values random or systematic? Cardinality sensible? Date formats consistent? Numeric ranges plausible (no negative ages, percentages over 100)?
+
+## Step 3: Choosing Your Tool
+
+| Scenario | Best Tool | Why |
+|---|---|---|
+| Quick exploration, single file | **pandas** | Fastest to write, rich API |
+| Large file (100MB+), multi-file joins | **DuckDB** | Columnar engine, minimal memory |
+| Existing database | **SQL** | Query where data lives |
+| Complex reshaping (pivot, melt) | **pandas** | Most flexible transformation API |
+| Window functions, CTEs | **DuckDB / SQL** | SQL is more expressive for these |
+
+**Rule of thumb:** Data fits in memory and one-off exploration? Use pandas. Joins, window functions, or files larger than RAM? Use DuckDB.
+
+```python
+import duckdb
+result = duckdb.sql("SELECT region, SUM(revenue) FROM 'sales.csv' GROUP BY region")
+result.df()  # convert to pandas DataFrame
+```
+
+## Step 4: Common Analysis Patterns
+
+### Aggregation and Grouping
+
+```python
+# pandas
+df.groupby("region")["revenue"].agg(["sum", "mean", "count"])
+```
+```sql
+-- SQL
+SELECT region, SUM(revenue) AS total, AVG(revenue) AS avg, COUNT(*) AS n
+FROM sales GROUP BY region ORDER BY total DESC;
+```
+
+### Ranking and Top-N
+
+```python
+df.nlargest(10, "revenue")
+```
+```sql
+SELECT *, RANK() OVER (PARTITION BY region ORDER BY revenue DESC) AS rnk
+FROM sales QUALIFY rnk <= 10;
+```
+
+### Time Series Aggregation
+
+```python
+df["date"] = pd.to_datetime(df["date"])
+df.set_index("date").resample("M")["revenue"].sum()
+```
+```sql
+SELECT DATE_TRUNC('month', order_date) AS month, SUM(revenue) AS total
+FROM orders GROUP BY month ORDER BY month;
+```
+
+### Joins
+
+```python
+merged = pd.merge(orders, customers, on="customer_id", how="left")
+```
+```sql
+SELECT o.*, c.segment, c.region
+FROM orders o LEFT JOIN customers c ON o.customer_id = c.customer_id;
+```
+
+### Window Functions (SQL)
+
+```sql
+-- Running total
+SUM(revenue) OVER (ORDER BY date ROWS UNBOUNDED PRECEDING)
+-- Month-over-month change
+revenue - LAG(revenue) OVER (ORDER BY month) AS mom_change
+-- Percentile rank
+PERCENT_RANK() OVER (PARTITION BY department ORDER BY salary)
+```
+
+### Pivot / Crosstab
+
+```python
+pd.pivot_table(df, values="revenue", index="region", columns="quarter", aggfunc="sum", fill_value=0)
+```
+```sql
+-- DuckDB
+PIVOT (SELECT region, category, revenue FROM sales)
+ON category USING SUM(revenue) GROUP BY region;
+```
+
+## Step 5: Visualization Guidance
+
+| Message | Chart Type |
+|---|---|
+| Comparison across categories | Bar chart (horizontal if many labels) |
+| Trend over time | Line chart |
+| Part-of-whole composition | Stacked bar or pie (2-5 slices only) |
+| Distribution | Histogram or box plot |
+| Correlation | Scatter plot |
+
+**Rules:**
+- Title states the insight, not just the metric. "Revenue grew 40% in Q2" not "Revenue by Quarter."
+- Label axes with units. "Revenue ($M)" not "Revenue."
+- Sort bar charts by value unless order is inherent (months, stages).
+- Avoid 3D charts, dual axes, and pie charts with more than 5 slices.
+
+```python
+import matplotlib.pyplot as plt
+import seaborn as sns
+sns.set_style("whitegrid")
+fig, ax = plt.subplots(figsize=(10, 6))
+ax.set_title("Clear, Descriptive Title")
+ax.set_xlabel("X Label with Units"); ax.set_ylabel("Y Label with Units")
+plt.tight_layout(); plt.savefig("output.png", dpi=150)
+```
+
+## Step 6: Output and Reporting
+
+Present results in the format most useful to the audience:
+- **Tables** — For precise values. Markdown for small results, CSV export for large.
+- **Charts** — For trends, comparisons, distributions. Save as PNG.
+- **Written narrative** — For executive audiences. Finding, evidence, caveats.
+
+### Reporting Structure
+
+Every analysis report should include: **Key Findings** (headline results), **Methodology** (data source, time range, filters), **Detailed Results** (tables/charts), **Caveats** (missing data, assumptions), and **Recommendations** (next steps).
+
+### Quality Checklist
+- [ ] Numbers add up — totals match, percentages sum correctly
+- [ ] Null handling is explicit — excluded, filled, or counted separately?
+- [ ] Date ranges are correct — no boundary date errors
+- [ ] Units are consistent — no mixing dollars/cents, bytes/megabytes
+- [ ] Sample size is noted — context for statistical significance
+- [ ] Results are reproducible — steps are clear enough to replicate

package/skills/deep-research/SKILL.md

@@ -0,0 +1,198 @@
+---
+name: deep-research
+description: Conduct systematic multi-angle web research across multiple sources and depths. Use when answering questions requiring current comprehensive information, researching topics in depth, or gathering data before content generation tasks.
+---
+
+# Deep Research
+
+A systematic methodology for conducting thorough web research across multiple angles, depths, and sources. Load this skill BEFORE starting any content generation task to ensure comprehensive information gathering.
+
+## When to Use
+
+**Always load this skill when:**
+
+### Research Questions
+- User asks "what is X", "explain X", "research X", "investigate X"
+- User wants to understand a concept, technology, or topic in depth
+- The question requires current, comprehensive information from multiple sources
+- A single web search would be insufficient to answer properly
+
+### Content Generation (Pre-research)
+- Creating presentations, articles, reports, or documentation
+- Creating frontend designs or UI mockups
+- Producing any content that requires real-world information, examples, or current data
+
+## When NOT to Use
+
+- **Academic literature specifically** — use the `academic-search` skill for scholarly papers
+- **GitHub repository analysis** — use the `github-deep-research` skill
+- **Questions answerable from the codebase** — read the code directly
+- **Consulting-grade reports** — use the `consulting-analysis` skill (which uses deep-research as a sub-step)
+
+## Core Principle
+
+**Never generate content based solely on general knowledge.** The quality of your output directly depends on the quality and quantity of research conducted beforehand. A single search query is NEVER enough.
+
+## Research Methodology
+
+### Phase 1: Broad Exploration
+
+Start with broad searches to understand the landscape:
+
+1. **Initial Survey**: Search for the main topic to understand the overall context
+2. **Identify Dimensions**: From initial results, identify key subtopics, themes, angles, or aspects that need deeper exploration
+3. **Map the Territory**: Note different perspectives, stakeholders, or viewpoints that exist
+
+Example:
+```
+Topic: "AI in healthcare"
+Initial searches:
+- "AI healthcare applications 2025"
+- "artificial intelligence medical diagnosis"
+- "healthcare AI market trends"
+
+Identified dimensions:
+- Diagnostic AI (radiology, pathology)
+- Treatment recommendation systems
+- Administrative automation
+- Patient monitoring
+- Regulatory landscape
+- Ethical considerations
+```
+
+### Phase 2: Deep Dive
+
+For each important dimension identified, conduct targeted research:
+
+1. **Specific Queries**: Use WebSearch with precise keywords for each subtopic
+2. **Multiple Phrasings**: Try different keyword combinations and phrasings
+3. **Fetch Full Content**: Use WebFetch to read important sources in full, not just snippets
+4. **Follow References**: When sources mention other important resources, search for those too
+
+Example:
+```
+Dimension: "Diagnostic AI in radiology"
+Targeted searches:
+- "AI radiology FDA approved systems"
+- "chest X-ray AI detection accuracy"
+- "radiology AI clinical trials results"
+
+Then fetch and read:
+- Key research papers or summaries
+- Industry reports
+- Real-world case studies
+```
+
+### Phase 3: Diversity & Validation
+
+Ensure comprehensive coverage by seeking diverse information types:
+
+| Information Type | Purpose | Example Searches |
+|-----------------|---------|------------------|
+| **Facts & Data** | Concrete evidence | "statistics", "data", "numbers", "market size" |
+| **Examples & Cases** | Real-world applications | "case study", "example", "implementation" |
+| **Expert Opinions** | Authority perspectives | "expert analysis", "interview", "commentary" |
+| **Trends & Predictions** | Future direction | "trends 2025", "forecast", "future of" |
+| **Comparisons** | Context and alternatives | "vs", "comparison", "alternatives" |
+| **Challenges & Criticisms** | Balanced view | "challenges", "limitations", "criticism" |
+
+### Phase 4: Synthesis Check
+
+Before proceeding to content generation, verify:
+
+- [ ] Have I searched from at least 3-5 different angles?
+- [ ] Have I fetched and read the most important sources in full?
+- [ ] Do I have concrete data, examples, and expert perspectives?
+- [ ] Have I explored both positive aspects and challenges/limitations?
+- [ ] Is my information current and from authoritative sources?
+
+**If any answer is NO, continue researching before generating content.**
+
+## Search Strategy Tips
+
+### Effective Query Patterns
+
+```
+# Be specific with context
+Bad:  "AI trends"
+Good: "enterprise AI adoption trends 2025"
+
+# Include authoritative source hints
+"[topic] research paper"
+"[topic] McKinsey report"
+"[topic] industry analysis"
+
+# Search for specific content types
+"[topic] case study"
+"[topic] statistics"
+"[topic] expert interview"
+
+# Use temporal qualifiers with the actual current year
+"[topic] 2025"
+"[topic] latest"
+"[topic] recent developments"
+```
+
+### Temporal Awareness
+
+**Always use today's date when forming time-sensitive search queries.** The current date is available in your system prompt context.
+
+Use the right level of precision depending on what the user is asking:
+
+| User intent | Temporal precision needed | Example query |
+|---|---|---|
+| "today / this morning / just released" | **Month + Day** | `"tech news February 28 2025"` |
+| "this week" | **Week range** | `"technology releases week of Feb 24 2025"` |
+| "recently / latest / new" | **Month** | `"AI breakthroughs February 2025"` |
+| "this year / trends" | **Year** | `"software trends 2025"` |
+
+**Rules:**
+- When the user asks about "today" or "just released", use **month + day + year** in your search queries to get same-day results
+- Never drop to year-only when day-level precision is needed — `"tech news 2025"` will NOT surface today's news
+- Try multiple phrasings: numeric form (`2025-02-28`), written form (`February 28 2025`), and relative terms (`today`, `this week`) across different queries
+
+### When to Use WebFetch
+
+Use WebFetch to read full page content when:
+- A search result looks highly relevant and authoritative
+- You need detailed information beyond the snippet
+- The source contains data, case studies, or expert analysis
+- You want to understand the full context of a finding
+
+### Iterative Refinement
+
+Research is iterative. After initial searches:
+1. Review what you have learned
+2. Identify gaps in your understanding
+3. Formulate new, more targeted queries
+4. Repeat until you have comprehensive coverage
+
+## Quality Bar
+
+Your research is sufficient when you can confidently answer:
+- What are the key facts and data points?
+- What are 2-3 concrete real-world examples?
+- What do experts say about this topic?
+- What are the current trends and future directions?
+- What are the challenges or limitations?
+- What makes this topic relevant or important now?
+
+## Common Mistakes to Avoid
+
+- Stopping after 1-2 searches
+- Relying on search snippets without reading full sources
+- Searching only one aspect of a multi-faceted topic
+- Ignoring contradicting viewpoints or challenges
+- Using outdated information when current data exists
+- Starting content generation before research is complete
+
+## Output
+
+After completing research, you should have:
+1. A comprehensive understanding of the topic from multiple angles
+2. Specific facts, data points, and statistics
+3. Real-world examples and case studies
+4. Expert perspectives and authoritative sources
+5. Current trends and relevant context
+
+**Only then proceed to content generation**, using the gathered information to create high-quality, well-informed content.

package/skills/docx/SKILL.md

@@ -0,0 +1,211 @@
+---
+name: docx
+description: Create, edit, and analyze Word documents (.docx) with support for tracked changes, comments, formatting preservation, and text extraction. Use when working with professional documents for creating, modifying, reviewing with redlines, or extracting content.
+---
+
+# DOCX creation, editing, and analysis
+
+## Overview
+
+A user may ask you to create, edit, or analyze the contents of a .docx file. A .docx file is essentially a ZIP archive containing XML files and other resources that you can read or edit. You have different tools and workflows available for different tasks.
+
+## When to Use
+
+- Creating new Word documents from scratch
+- Editing or reviewing existing .docx files with tracked changes
+- Extracting text, comments, or metadata from Word documents
+- Adding redline/tracked changes to legal, business, or academic documents
+- Converting documents between formats
+
+## When NOT to Use
+
+- **Spreadsheets** — use the `xlsx` skill
+- **Presentations** — use the `pptx` skill
+- **PDF documents** — use the `pdf` skill
+- **Plain text or Markdown** — edit directly, no special tooling needed
+
+## Workflow Decision Tree
+
+### Reading/Analyzing Content
+Use "Text extraction" or "Raw XML access" sections below
+
+### Creating New Document
+Use "Creating a new Word document" workflow
+
+### Editing Existing Document
+- **Your own document + simple changes**
+  Use "Basic OOXML editing" workflow
+
+- **Someone else's document**
+  Use **"Redlining workflow"** (recommended default)
+
+- **Legal, academic, business, or government docs**
+  Use **"Redlining workflow"** (required)
+
+## Reading and analyzing content
+
+### Text extraction
+If you just need to read the text contents of a document, you should convert the document to markdown using pandoc. Pandoc provides excellent support for preserving document structure and can show tracked changes:
+
+```bash
+# Convert document to markdown with tracked changes
+pandoc --track-changes=all path-to-file.docx -o output.md
+# Options: --track-changes=accept/reject/all
+```
+
+### Raw XML access
+You need raw XML access for: comments, complex formatting, document structure, embedded media, and metadata. For any of these features, you'll need to unpack a document and read its raw XML contents.
+
+#### Unpacking a file
+`python ooxml/scripts/unpack.py <office_file> <output_directory>`
+
+#### Key file structures
+* `word/document.xml` - Main document contents
+* `word/comments.xml` - Comments referenced in document.xml
+* `word/media/` - Embedded images and media files
+* Tracked changes use `<w:ins>` (insertions) and `<w:del>` (deletions) tags
+
+## Creating a new Word document
+
+When creating a new Word document from scratch, use **docx-js**, which allows you to create Word documents using JavaScript/TypeScript.
+
+### Workflow
+1. **MANDATORY - READ ENTIRE FILE**: Read [`docx-js.md`](docx-js.md) (~500 lines) completely from start to finish. **NEVER set any range limits when reading this file.** Read the full file content for detailed syntax, critical formatting rules, and best practices before proceeding with document creation.
+2. Create a JavaScript/TypeScript file using Document, Paragraph, TextRun components (You can assume all dependencies are installed, but if not, refer to the dependencies section below)
+3. Export as .docx using Packer.toBuffer()
+
+## Editing an existing Word document
+
+When editing an existing Word document, use the **Document library** (a Python library for OOXML manipulation). The library automatically handles infrastructure setup and provides methods for document manipulation. For complex scenarios, you can access the underlying DOM directly through the library.
+
+### Workflow
+1. **MANDATORY - READ ENTIRE FILE**: Read [`ooxml.md`](ooxml.md) (~600 lines) completely from start to finish. **NEVER set any range limits when reading this file.** Read the full file content for the Document library API and XML patterns for directly editing document files.
+2. Unpack the document: `python ooxml/scripts/unpack.py <office_file> <output_directory>`
+3. Create and run a Python script using the Document library (see "Document Library" section in ooxml.md)
+4. Pack the final document: `python ooxml/scripts/pack.py <input_directory> <office_file>`
+
+The Document library provides both high-level methods for common operations and direct DOM access for complex scenarios.
+
+## Redlining workflow for document review
+
+This workflow allows you to plan comprehensive tracked changes using markdown before implementing them in OOXML. **CRITICAL**: For complete tracked changes, you must implement ALL changes systematically.
+
+**Batching Strategy**: Group related changes into batches of 3-10 changes. This makes debugging manageable while maintaining efficiency. Test each batch before moving to the next.
+
+**Principle: Minimal, Precise Edits**
+When implementing tracked changes, only mark text that actually changes. Repeating unchanged text makes edits harder to review and appears unprofessional. Break replacements into: [unchanged text] + [deletion] + [insertion] + [unchanged text]. Preserve the original run's RSID for unchanged text by extracting the `<w:r>` element from the original and reusing it.
+
+Example - Changing "30 days" to "60 days" in a sentence:
+```python
+# BAD - Replaces entire sentence
+'<w:del><w:r><w:delText>The term is 30 days.</w:delText></w:r></w:del><w:ins><w:r><w:t>The term is 60 days.</w:t></w:r></w:ins>'
+
+# GOOD - Only marks what changed, preserves original <w:r> for unchanged text
+'<w:r w:rsidR="00AB12CD"><w:t>The term is </w:t></w:r><w:del><w:r><w:delText>30</w:delText></w:r></w:del><w:ins><w:r><w:t>60</w:t></w:r></w:ins><w:r w:rsidR="00AB12CD"><w:t> days.</w:t></w:r>'
+```
+
+### Tracked changes workflow
+
+1. **Get markdown representation**: Convert document to markdown with tracked changes preserved:
+   ```bash
+   pandoc --track-changes=all path-to-file.docx -o current.md
+   ```
+
+2. **Identify and group changes**: Review the document and identify ALL changes needed, organizing them into logical batches:
+
+   **Location methods** (for finding changes in XML):
+   - Section/heading numbers (e.g., "Section 3.2", "Article IV")
+   - Paragraph identifiers if numbered
+   - Grep patterns with unique surrounding text
+   - Document structure (e.g., "first paragraph", "signature block")
+   - **DO NOT use markdown line numbers** - they don't map to XML structure
+
+   **Batch organization** (group 3-10 related changes per batch):
+   - By section: "Batch 1: Section 2 amendments", "Batch 2: Section 5 updates"
+   - By type: "Batch 1: Date corrections", "Batch 2: Party name changes"
+   - By complexity: Start with simple text replacements, then tackle complex structural changes
+   - Sequential: "Batch 1: Pages 1-3", "Batch 2: Pages 4-6"
+
+3. **Read documentation and unpack**:
+   - **MANDATORY - READ ENTIRE FILE**: Read [`ooxml.md`](ooxml.md) (~600 lines) completely from start to finish. **NEVER set any range limits when reading this file.** Pay special attention to the "Document Library" and "Tracked Change Patterns" sections.
+   - **Unpack the document**: `python ooxml/scripts/unpack.py <file.docx> <dir>`
+   - **Note the suggested RSID**: The unpack script will suggest an RSID to use for your tracked changes. Copy this RSID for use in step 4b.
+
+4. **Implement changes in batches**: Group changes logically (by section, by type, or by proximity) and implement them together in a single script. This approach:
+   - Makes debugging easier (smaller batch = easier to isolate errors)
+   - Allows incremental progress
+   - Maintains efficiency (batch size of 3-10 changes works well)
+
+   **Suggested batch groupings:**
+   - By document section (e.g., "Section 3 changes", "Definitions", "Termination clause")
+   - By change type (e.g., "Date changes", "Party name updates", "Legal term replacements")
+   - By proximity (e.g., "Changes on pages 1-3", "Changes in first half of document")
+
+   For each batch of related changes:
+
+   **a. Map text to XML**: Grep for text in `word/document.xml` to verify how text is split across `<w:r>` elements.
+
+   **b. Create and run script**: Use `get_node` to find nodes, implement changes, then `doc.save()`. See **"Document Library"** section in ooxml.md for patterns.
+
+   **Note**: Always grep `word/document.xml` immediately before writing a script to get current line numbers and verify text content. Line numbers change after each script run.
+
+5. **Pack the document**: After all batches are complete, convert the unpacked directory back to .docx:
+   ```bash
+   python ooxml/scripts/pack.py unpacked reviewed-document.docx
+   ```
+
+6. **Final verification**: Do a comprehensive check of the complete document:
+   - Convert final document to markdown:
+     ```bash
+     pandoc --track-changes=all reviewed-document.docx -o verification.md
+     ```
+   - Verify ALL changes were applied correctly:
+     ```bash
+     grep "original phrase" verification.md  # Should NOT find it
+     grep "replacement phrase" verification.md  # Should find it
+     ```
+   - Check that no unintended changes were introduced
+
+
+## Converting Documents to Images
+
+To visually analyze Word documents, convert them to images using a two-step process:
+
+1. **Convert DOCX to PDF**:
+   ```bash
+   soffice --headless --convert-to pdf document.docx
+   ```
+
+2. **Convert PDF pages to JPEG images**:
+   ```bash
+   pdftoppm -jpeg -r 150 document.pdf page
+   ```
+   This creates files like `page-1.jpg`, `page-2.jpg`, etc.
+
+Options:
+- `-r 150`: Sets resolution to 150 DPI (adjust for quality/size balance)
+- `-jpeg`: Output JPEG format (use `-png` for PNG if preferred)
+- `-f N`: First page to convert (e.g., `-f 2` starts from page 2)
+- `-l N`: Last page to convert (e.g., `-l 5` stops at page 5)
+- `page`: Prefix for output files
+
+Example for specific range:
+```bash
+pdftoppm -jpeg -r 150 -f 2 -l 5 document.pdf page  # Converts only pages 2-5
+```
+
+## Code Style Guidelines
+**IMPORTANT**: When generating code for DOCX operations:
+- Write concise code
+- Avoid verbose variable names and redundant operations
+- Avoid unnecessary print statements
+
+## Dependencies
+
+Required dependencies (install if not available):
+
+- **pandoc**: `sudo apt-get install pandoc` (for text extraction)
+- **docx**: `npm install -g docx` (for creating new documents)
+- **LibreOffice**: `sudo apt-get install libreoffice` (for PDF conversion)
+- **Poppler**: `sudo apt-get install poppler-utils` (for pdftoppm to convert PDF to images)
+- **defusedxml**: `pip install defusedxml` (for secure XML parsing)