@wentorai/research-plugins 1.0.0 → 1.2.0

package/skills/research/automation/kedro-pipeline-guide/SKILL.md

@@ -0,0 +1,216 @@
+---
+name: kedro-pipeline-guide
+description: "Build reproducible data science pipelines with Kedro for research projects"
+metadata:
+  openclaw:
+    emoji: "🔧"
+    category: "research"
+    subcategory: "automation"
+    keywords: ["pipeline", "reproducibility", "data-science", "workflow", "automation", "mlops"]
+    source: "https://github.com/kedro-org/kedro"
+---
+
+# Kedro Pipeline Guide
+
+## Overview
+
+Kedro is an open-source Python framework for creating reproducible, maintainable, and modular data science pipelines. Developed originally at McKinsey's QuantumBlack labs, Kedro provides an opinionated project structure and a set of conventions that transform ad-hoc analysis scripts into production-quality code that can be tested, versioned, and shared across research teams.
+
+In academic research, reproducibility is both a scientific imperative and a practical challenge. Jupyter notebooks and standalone scripts often become tangled webs of dependencies that are difficult to re-run months later when responding to reviewer comments or extending prior work. Kedro addresses this by separating data processing logic from data access, enforcing explicit pipeline definitions, and providing built-in data versioning and experiment tracking.
+
+With over 11,000 GitHub stars, Kedro has gained adoption across industry and academia. Its design philosophy aligns naturally with the needs of computational research: clear data lineage, parameterized experiments, and the ability to scale from a laptop to a cluster without rewriting code.
+
+## Installation and Setup
+
+Install Kedro via pip:
+
+```bash
+pip install kedro
+```
+
+Create a new project using the Kedro starter:
+
+```bash
+kedro new --name my-research-project --tools lint,test,docs
+cd my-research-project
+```
+
+This generates a standardized project structure:
+
+```
+my-research-project/
+  conf/
+    base/
+      catalog.yml        # Data source definitions
+      parameters.yml     # Experiment parameters
+    local/               # Local overrides (gitignored)
+  src/
+    my_research_project/
+      pipelines/
+        data_processing/
+          nodes.py       # Pure Python functions
+          pipeline.py    # Pipeline definition
+        modeling/
+          nodes.py
+          pipeline.py
+      pipeline_registry.py
+  data/                  # Local data directory
+  notebooks/             # Jupyter notebooks
+  tests/                 # Unit tests
+```
+
+Install project dependencies:
+
+```bash
+pip install -e ".[dev]"
+```
+
+## Core Concepts
+
+**Nodes**: The fundamental units of computation in Kedro. Each node is a pure Python function with explicitly declared inputs and outputs:
+
+```python
+# src/my_research_project/pipelines/data_processing/nodes.py
+
+import pandas as pd
+from sklearn.preprocessing import StandardScaler
+
+def clean_raw_data(raw_data: pd.DataFrame) -> pd.DataFrame:
+    """Remove missing values and outliers from raw experimental data."""
+    cleaned = raw_data.dropna(subset=["measurement", "condition"])
+    q1 = cleaned["measurement"].quantile(0.01)
+    q99 = cleaned["measurement"].quantile(0.99)
+    return cleaned[cleaned["measurement"].between(q1, q99)]
+
+def normalize_features(
+    cleaned_data: pd.DataFrame, parameters: dict
+) -> pd.DataFrame:
+    """Standardize feature columns specified in parameters."""
+    feature_cols = parameters["feature_columns"]
+    scaler = StandardScaler()
+    result = cleaned_data.copy()
+    result[feature_cols] = scaler.fit_transform(cleaned_data[feature_cols])
+    return result
+```
+
+**Pipelines**: Chains of nodes connected through named datasets:
+
+```python
+# src/my_research_project/pipelines/data_processing/pipeline.py
+
+from kedro.pipeline import Pipeline, node, pipeline
+from .nodes import clean_raw_data, normalize_features
+
+def create_pipeline(**kwargs) -> Pipeline:
+    return pipeline([
+        node(
+            func=clean_raw_data,
+            inputs="raw_experiment_data",
+            outputs="cleaned_data",
+            name="clean_data_node",
+        ),
+        node(
+            func=normalize_features,
+            inputs=["cleaned_data", "params:preprocessing"],
+            outputs="normalized_data",
+            name="normalize_node",
+        ),
+    ])
+```
+
+**Data Catalog**: A declarative registry that maps logical dataset names to physical storage:
+
+```yaml
+# conf/base/catalog.yml
+raw_experiment_data:
+  type: pandas.CSVDataset
+  filepath: data/01_raw/experiment_results.csv
+
+cleaned_data:
+  type: pandas.ParquetDataset
+  filepath: data/02_intermediate/cleaned.parquet
+
+normalized_data:
+  type: pandas.ParquetDataset
+  filepath: data/03_primary/normalized.parquet
+  versioned: true
+```
+
+The `versioned: true` flag automatically creates timestamped versions of outputs, enabling exact reproduction of prior runs.
+
+**Parameters**: Experiment configuration separated from code:
+
+```yaml
+# conf/base/parameters.yml
+preprocessing:
+  feature_columns:
+    - temperature
+    - pressure
+    - concentration
+  outlier_method: iqr
+
+modeling:
+  algorithm: random_forest
+  n_estimators: 500
+  max_depth: 10
+  test_size: 0.2
+  random_seed: 42
+```
+
+## Running and Visualizing Pipelines
+
+Execute the full pipeline:
+
+```bash
+kedro run
+```
+
+Run a specific pipeline or node:
+
+```bash
+kedro run --pipeline data_processing
+kedro run --nodes clean_data_node
+```
+
+Visualize the pipeline dependency graph:
+
+```bash
+pip install kedro-viz
+kedro viz run
+```
+
+This launches an interactive web visualization showing the complete data flow, making it easy to understand and communicate your analytical pipeline to collaborators and reviewers.
+
+## Research Workflow Integration
+
+**Experiment Reproducibility**: Every Kedro run uses explicit parameters and versioned data. Store parameter files in Git alongside code to create a complete record of every experiment configuration.
+
+**Reviewer Response**: When peer reviewers request additional analyses or modified parameters, change `parameters.yml` and re-run. The pipeline automatically reprocesses only affected downstream nodes.
+
+**Team Collaboration**: Multiple researchers can work on different pipeline modules simultaneously. The explicit input/output contracts between nodes prevent integration conflicts.
+
+**Scaling Computation**: Kedro pipelines can be deployed to distributed computing platforms without code changes using runners:
+
+```bash
+# Run with parallel execution
+kedro run --runner=ParallelRunner
+
+# Deploy to Airflow, Prefect, or other orchestrators
+pip install kedro-airflow
+kedro airflow create
+```
+
+**Integration with Jupyter**: Use Kedro notebooks for exploration while maintaining the pipeline for production runs:
+
+```bash
+kedro jupyter notebook
+```
+
+The Kedro Jupyter integration automatically loads the project catalog and parameters, bridging the gap between interactive exploration and pipeline execution.
+
+## References
+
+- Kedro repository: https://github.com/kedro-org/kedro
+- Kedro documentation: https://docs.kedro.org/
+- Kedro-Viz for pipeline visualization: https://github.com/kedro-org/kedro-viz
+- QuantumBlack blog on reproducible data science

package/skills/research/automation/mle-agent-guide/SKILL.md

@@ -0,0 +1,139 @@
+---
+name: mle-agent-guide
+description: "Intelligent companion for ML engineering with arXiv integration"
+version: 1.0.0
+author: wentor-community
+source: https://github.com/MLSys-Tools/MLE-agent
+metadata:
+  openclaw:
+    category: "research"
+    subcategory: "automation"
+    keywords:
+      - machine-learning
+      - ml-engineering
+      - arxiv-integration
+      - experiment-tracking
+      - model-development
+      - ai-engineering
+---
+
+# MLE Agent Guide
+
+A skill for using an intelligent ML engineering companion that integrates arXiv paper discovery with experiment implementation, tracking, and iteration. Based on MLE-agent (2K stars), this skill helps researchers bridge the gap between reading about new ML techniques and implementing them in their own projects.
+
+## Overview
+
+Machine learning research moves at an extraordinary pace, with hundreds of new papers appearing on arXiv daily. Researchers struggle not just to keep up with the literature but to translate promising ideas into working implementations. MLE-agent addresses this by combining paper discovery, technique extraction, implementation assistance, and experiment management into a unified workflow.
+
+This skill is designed for ML researchers and engineers who want to quickly prototype ideas from papers, systematically compare approaches, and maintain organized experiment records throughout the research process.
+
+## arXiv Integration
+
+The skill provides sophisticated arXiv paper discovery and analysis:
+
+**Paper Discovery**
+- Monitor arXiv categories relevant to the researcher's interests (cs.LG, cs.CL, cs.CV, stat.ML, etc.)
+- Filter new papers by keyword relevance, author familiarity, and citation velocity
+- Rank papers by potential applicability to current research projects
+- Generate daily or weekly digests of the most relevant new papers
+- Track papers from specific authors or research groups
+
+**Paper Analysis**
+- Extract the core technical contribution from each paper
+- Identify the proposed method's key components and hyperparameters
+- Compare the method against baselines reported in the paper
+- Note the datasets and evaluation metrics used
+- Assess reproducibility based on available code, data, and method description detail
+
+**Technique Extraction**
+- Distill the algorithmic steps from the paper's method section
+- Identify required input data formats and preprocessing steps
+- Map the technique's computational requirements (GPU memory, training time)
+- Note dependencies on specific frameworks or libraries
+- Extract training recipes including learning rate schedules, batch sizes, and augmentation strategies
+
+## Experiment Workflow
+
+The core experiment management workflow:
+
+**Project Setup**
+- Initialize a structured project directory with standard ML conventions
+- Set up experiment configuration files (YAML or JSON)
+- Configure logging and metric tracking infrastructure
+- Establish a baseline model and dataset pipeline
+- Create a reproducibility checklist (random seeds, library versions, hardware specs)
+
+**Implementation Assistance**
+- Translate extracted techniques from papers into implementation plans
+- Generate code scaffolds for new model architectures
+- Suggest appropriate loss functions and optimization strategies
+- Help debug common ML issues (gradient problems, data loading bottlenecks, memory issues)
+- Provide code review focused on ML best practices
+
+**Experiment Execution**
+- Configure hyperparameter search spaces based on paper recommendations
+- Set up systematic ablation studies to understand component contributions
+- Track metrics across training runs with consistent logging
+- Generate comparison tables and learning curve plots
+- Flag anomalous training behavior (loss spikes, metric plateaus, divergence)
+
+**Result Analysis**
+- Compare results across experiment runs with statistical tests
+- Identify which hyperparameters have the strongest effect on performance
+- Generate publication-ready result tables with confidence intervals
+- Produce ablation study summaries highlighting each component's contribution
+- Assess whether results match or deviate from paper-reported numbers
+
+## ML Engineering Best Practices
+
+The skill enforces ML engineering standards throughout the workflow:
+
+**Reproducibility**
+- Log all random seeds and ensure deterministic operations where possible
+- Record exact library versions in requirements files
+- Version control all configuration files alongside code
+- Store model checkpoints with their corresponding configurations
+- Document any manual steps in the pipeline
+
+**Code Quality**
+- Separate data loading, model definition, training, and evaluation into distinct modules
+- Use configuration files rather than hardcoded hyperparameters
+- Implement proper validation splits independent of test sets
+- Add assertions for tensor shapes and value ranges at module boundaries
+- Write unit tests for custom layers and data transformations
+
+**Resource Management**
+- Estimate computational costs before launching large experiments
+- Use gradient accumulation to train with limited GPU memory
+- Implement early stopping to avoid wasting compute on unpromising runs
+- Profile code to identify and optimize bottlenecks
+- Clean up intermediate artifacts and checkpoints to manage storage
+
+## Common ML Research Patterns
+
+The skill recognizes and supports common research patterns:
+
+**Baseline Comparison** - Implement and evaluate standard baselines before proposing improvements
+**Ablation Study** - Systematically remove or vary components to understand contributions
+**Scaling Analysis** - Test how performance changes with model size, data size, or compute
+**Transfer Learning** - Adapt pretrained models to new tasks with appropriate fine-tuning strategies
+**Ensemble Methods** - Combine multiple models for improved and more robust performance
+
+## Integration with Research-Claw
+
+This skill connects with the Research-Claw ecosystem:
+
+- Use literature search skills to find relevant papers for the current project
+- Feed experiment results to writing skills for paper drafting
+- Connect with analysis skills for statistical evaluation of results
+- Store successful experiment configurations as reusable templates
+- Share experiment logs and results with collaborators via the platform
+
+## Best Practices
+
+- Always establish baselines before implementing novel techniques
+- Read the paper's appendix and supplementary materials for implementation details often missing from the main text
+- Start with the paper's reported hyperparameters, then tune for your specific setup
+- Keep a research log documenting what was tried, what worked, and what failed
+- Verify your reimplementation on the paper's original dataset before applying to new data
+- Track compute costs alongside performance metrics to evaluate efficiency trade-offs

package/skills/research/automation/paper-to-agent-guide/SKILL.md

@@ -0,0 +1,116 @@
+---
+name: paper-to-agent-guide
+description: "Transform research papers into interactive AI agents for exploration"
+version: 1.0.0
+author: wentor-community
+source: https://github.com/paper2agent/Paper2Agent
+metadata:
+  openclaw:
+    category: "research"
+    subcategory: "automation"
+    keywords:
+      - paper-parsing
+      - agent-generation
+      - interactive-papers
+      - research-automation
+      - knowledge-extraction
+---
+
+# Paper-to-Agent Guide
+
+A skill for transforming published research papers into interactive AI agents that can answer questions, explain methodology, and help replicate findings. Based on Paper2Agent (2K stars), this skill guides the agent through extracting structured knowledge from academic papers and creating conversational interfaces for deep exploration.
+
+## Overview
+
+Traditional paper reading is linear and passive. Paper-to-Agent converts this into an active, queryable experience. By parsing a paper's structure, extracting key claims, methodology details, and results, the agent becomes an expert on that specific paper, ready to answer follow-up questions, explain complex sections, and connect findings to the broader literature.
+
+This approach is especially valuable for interdisciplinary researchers who need to quickly understand papers outside their primary expertise, for journal clubs seeking deeper discussion, and for students learning to critically evaluate published research.
+
+## Paper Parsing Workflow
+
+The agent should follow this structured workflow when converting a paper to an interactive agent:
+
+**Step 1: Structure Extraction**
+- Identify the paper's sections (abstract, introduction, methods, results, discussion, references)
+- Extract the title, authors, affiliations, and publication venue
+- Identify figure and table captions along with their referenced locations
+- Note supplementary materials and their availability
+- Detect the paper type (empirical, theoretical, review, meta-analysis)
+
+**Step 2: Claim Extraction**
+- Identify the primary research question or hypothesis
+- Extract all major claims made in the paper
+- Map each claim to its supporting evidence (data, citations, arguments)
+- Note the strength of evidence for each claim (strong, moderate, suggestive)
+- Identify limitations acknowledged by the authors
+
+**Step 3: Methodology Mapping**
+- Document the complete experimental or analytical pipeline
+- Extract parameter values, dataset descriptions, and evaluation metrics
+- Identify software tools and libraries used
+- Note any preprocessing or data cleaning steps
+- Map the methodology to established frameworks in the field
+
+## Interactive Exploration Capabilities
+
+Once a paper has been parsed, the agent can support these interaction patterns:
+
+**Question-Answering**
+- Answer specific questions about the paper's content with source references
+- Explain technical terms in context of how the paper uses them
+- Compare the paper's approach to common alternatives
+- Identify what the paper does and does not address
+- Generate summaries at different levels of detail (tweet-length, abstract, detailed)
+
+**Critical Analysis**
+- Evaluate the validity of statistical analyses
+- Identify potential confounds not addressed by the authors
+- Assess whether conclusions follow from the presented evidence
+- Compare results to related work in the field
+- Suggest follow-up experiments that would strengthen the findings
+
+**Replication Assistance**
+- Generate step-by-step replication guides from the methods section
+- Identify missing details needed for exact replication
+- Suggest parameter ranges for robustness checks
+- Create data collection templates based on the paper's design
+- List required resources (compute, data, equipment) for replication
+
+## Knowledge Graph Construction
+
+The skill supports building knowledge graphs from processed papers:
+
+- Extract entities (methods, datasets, metrics, tools, concepts)
+- Map relationships between entities (uses, extends, contradicts, supports)
+- Link to external knowledge bases (Semantic Scholar, OpenAlex, DOI)
+- Track citation chains for key claims
+- Identify research lineages and methodological evolution
+
+## Multi-Paper Analysis
+
+When multiple papers have been processed, the agent can:
+
+- Compare methodologies across papers addressing similar questions
+- Identify consensus findings and areas of disagreement
+- Trace the evolution of a research direction over time
+- Build synthesis summaries combining evidence from multiple sources
+- Detect gaps in the literature that no existing paper addresses
+
+## Integration with Research-Claw
+
+This skill connects with other Research-Claw capabilities:
+
+- Use literature search skills to find papers for processing
+- Feed extracted knowledge into writing skills for literature reviews
+- Connect methodology details to analysis skills for replication
+- Store parsed papers in the local knowledge base for future reference
+- Generate citation entries compatible with reference management tools
+
+## Practical Tips
+
+- Start with the abstract and conclusion to determine if full parsing is worthwhile
+- Focus deep extraction on methods and results sections for empirical papers
+- For theoretical papers, prioritize definitions, theorems, and proof sketches
+- Always verify extracted claims against the original text before presenting them
+- Flag areas where the paper's writing is ambiguous or inconsistent
+- Use the parsed representation to generate discussion questions for journal clubs