@anysiteio/agent-skills 1.0.1 → 1.2.0

package/skills/anysite-cli/references/dataset-guide.md

@@ -0,0 +1,434 @@
+# Dataset Pipeline & Database Guide
+
+## Dataset YAML Configuration
+
+### Full Structure
+
+```yaml
+name: my-dataset
+description: Optional description
+
+sources:
+  - id: source_id              # Unique identifier
+    endpoint: /api/...         # API endpoint path
+    params: {}                 # Static API parameters
+    from_file: inputs.txt      # Input file (txt/JSONL/CSV)
+    file_field: column_name    # CSV column to extract
+    input_key: param_name      # API parameter for input values
+    input_template: {}         # Transform values before API call
+    dependency:                # Link to parent source
+      from_source: parent_id
+      field: urn.value         # Dot-notation field to extract
+      match_by: name           # Alternative: fuzzy match by name
+      dedupe: true             # Remove duplicate values
+    parallel: 3                # Concurrent requests
+    rate_limit: "10/s"         # Rate limiting
+    on_error: skip             # stop or skip
+    transform:                 # Post-collection transform (for exports only)
+      filter: '.count > 10'   # Safe filter expression
+      fields: [name, url]     # Field selection with aliases
+      add_columns:             # Static columns to inject
+        batch: "q1-2026"
+    export:                    # Per-source export destinations
+      - type: file
+        path: ./output/{{source}}-{{date}}.csv
+        format: csv            # json, jsonl, csv
+      - type: webhook
+        url: https://example.com/hook
+        headers: {X-Token: abc}
+    db_load:                   # Database loading config
+      table: custom_name       # Override table name
+      fields: [name, url]      # Fields to include
+      exclude: [_input_value]  # Fields to exclude
+
+storage:
+  format: parquet
+  path: ./data/
+  partition_by: [source_id, collected_date]
+
+schedule:                      # Collection schedule
+  cron: "0 9 * * *"           # Standard cron expression
+
+notifications:                 # Webhook notifications
+  on_complete:
+    - url: "https://hooks.slack.com/xxx"
+      headers: {X-Token: abc}
+  on_failure:
+    - url: "https://alerts.example.com/fail"
+```
+
+### Three Source Types
+
+**Independent** — single API call with static params:
+```yaml
+- id: search_results
+  endpoint: /api/linkedin/search/users
+  params:
+    keywords: "software engineer"
+    count: 50
+```
+
+**from_file** — iterate over values from a file:
+```yaml
+- id: companies
+  endpoint: /api/linkedin/company
+  from_file: companies.txt      # One value per line
+  input_key: company             # API parameter name
+  parallel: 3
+```
+
+**Dependent** — extract values from parent source output:
+```yaml
+- id: employees
+  endpoint: /api/linkedin/company/employees
+  dependency:
+    from_source: companies       # Parent source ID
+    field: urn.value             # Dot-notation path in parent records
+    dedupe: true                 # Deduplicate extracted values
+  input_key: companies           # API parameter name
+  input_template:                # Transform value before API call
+    companies:
+      - type: company
+        value: "{value}"         # {value} is replaced with extracted value
+    count: 5
+```
+
+### Dependency Chains
+
+Sources are topologically sorted — parents always run before children. Multi-level chains work automatically:
+
+```
+companies → employees → profiles → posts → comments
+```
+
+### input_template
+
+Transforms extracted values before passing to the API. Use `{value}` placeholder:
+
+```yaml
+input_template:
+  urn: "urn:li:fsd_profile:{value}"
+  count: 5
+```
+
+If `field: urn.value` extracts `"ACoAABCDEF"`, the API receives:
+```json
+{"urn": "urn:li:fsd_profile:ACoAABCDEF", "count": 5}
+```
+
+### Dot-Notation Field Extraction
+
+When Parquet stores nested objects as JSON strings, dot-notation traverses them:
+
+- `urn.value` — parses JSON string in `urn` field, extracts `.value`
+- `experience[0].company_urn` — array index + nested field
+- `internal_id.value` — nested object access
+
+---
+
+## Collection Commands
+
+```bash
+# Preview collection plan with estimated request counts
+anysite dataset collect dataset.yaml --dry-run
+
+# Full collection
+anysite dataset collect dataset.yaml
+
+# Collect and auto-load into database
+anysite dataset collect dataset.yaml --load-db pg
+
+# Incremental — skip inputs already collected
+anysite dataset collect dataset.yaml --incremental --load-db pg
+
+# Single source + its dependencies
+anysite dataset collect dataset.yaml --source employees
+
+# Quiet mode
+anysite dataset collect dataset.yaml --quiet
+```
+
+### Provenance Tracking
+
+Dependent and from_file records automatically get metadata columns:
+- `_input_value` — the raw extracted value that produced this record
+- `_parent_source` — parent source ID (dependent sources only)
+
+These enable FK linking when loading into a database.
+
+### Incremental Collection
+
+With `--incremental`:
+1. Independent sources: skipped if already collected today
+2. Dependent/from_file sources: skips individual input values already in `metadata.json`
+3. New values are still collected and tracked
+
+### Storage Layout
+
+```
+<storage.path>/
+  raw/<source_id>/<YYYY-MM-DD>.parquet
+  metadata.json
+```
+
+---
+
+## Query Commands (DuckDB)
+
+Each source becomes a DuckDB view named after its ID.
+
+```bash
+# Direct SQL
+anysite dataset query dataset.yaml --sql "SELECT * FROM companies LIMIT 10"
+
+# Shorthand: --source auto-generates SELECT
+anysite dataset query dataset.yaml --source profiles
+
+# Dot-notation field extraction
+anysite dataset query dataset.yaml --source profiles \
+  --fields "name, urn.value AS urn_id, headline"
+# Generates: SELECT name, json_extract_string(urn, '$.value') AS urn_id, headline FROM profiles
+
+# Output options
+anysite dataset query dataset.yaml --sql "SELECT * FROM companies" \
+  --format csv --output companies.csv
+
+# Interactive SQL shell
+anysite dataset query dataset.yaml --interactive
+
+# Column statistics
+anysite dataset stats dataset.yaml --source companies
+
+# Data profiling (completeness, record counts)
+anysite dataset profile dataset.yaml
+```
+
+---
+
+## Database Loading (load-db)
+
+Load Parquet data into a relational database with automatic FK linking.
+
+```bash
+anysite dataset load-db dataset.yaml -c <connection_name> [OPTIONS]
+```
+
+### Options
+```
+--connection, -c TEXT    Database connection name (required)
+--source, -s TEXT        Load specific source + dependencies
+--drop-existing          Drop tables before creating
+--dry-run                Show plan without executing
+--quiet, -q              Suppress output
+```
+
+### What load-db Does
+
+1. Reads Parquet files for each source (in topological order)
+2. Infers SQL schema from data (integer, float, text, json, etc.)
+3. Creates tables with auto-increment `id` primary key
+4. Inserts rows, tracking which `_input_value` maps to which `id`
+5. For child sources: adds `{parent_source}_id` FK column using provenance
+
+### db_load Config
+
+Control which fields go to the database per source:
+
+```yaml
+db_load:
+  table: people                    # Custom table name (default: source ID)
+  fields:                          # Explicit field list
+    - name
+    - url
+    - urn.value AS urn_id          # Dot-notation with alias
+    - experience                   # JSON columns stored as TEXT/JSONB
+  exclude:                         # Fields to skip (default: _input_value, _parent_source)
+    - _input_value
+    - _parent_source
+    - raw_html
+```
+
+Without `db_load`: all fields except `_input_value` and `_parent_source` are loaded.
+
+### FK Linking Example
+
+Given: companies → employees → posts
+
+Result in database:
+- `companies` table: `id`, name, url, ...
+- `employees` table: `id`, name, ..., `companies_id` (FK to companies.id)
+- `posts` table: `id`, text, ..., `employees_id` (FK to employees.id)
+
+---
+
+## Database Commands (anysite db)
+
+### Connection Management
+```bash
+anysite db add <name>              # Interactive add
+anysite db list                    # List all connections
+anysite db test <name>             # Test connectivity
+anysite db info <name>             # Show connection details
+anysite db remove <name>           # Delete connection
+```
+
+Connections stored in `~/.anysite/connections.yaml`. Passwords use env var references for security.
+
+### Schema Inspection
+```bash
+anysite db schema <name>                  # List all tables
+anysite db schema <name> --table users    # Show columns
+```
+
+### Data Operations
+```bash
+# Insert from stdin (auto-create table)
+cat data.jsonl | anysite db insert <name> --table users --stdin --auto-create
+
+# Insert from file
+anysite db insert <name> --table users --file data.jsonl
+
+# Upsert (update on conflict)
+anysite db upsert <name> --table users --conflict-columns id --stdin
+
+# Insert with conflict handling
+anysite db insert <name> --table users --file data.jsonl \
+  --on-conflict ignore --conflict-columns email
+```
+
+### SQL Queries
+```bash
+anysite db query <name> --sql "SELECT * FROM users" --format table
+anysite db query <name> --file report.sql --format csv --output report.csv
+```
+
+### Supported Databases
+- **SQLite** — `--type sqlite --path ./data.db`
+- **PostgreSQL** — `--type postgres --host localhost --database mydb --user app --password-env DB_PASS`
+- PostgreSQL also supports `--url-env DATABASE_URL` for connection strings
+
+---
+
+## Per-Source Transform
+
+Transforms apply to export destinations only. Parquet always stores full records (needed for dependency resolution).
+
+```yaml
+transform:
+  filter: '.employee_count > 10 and .status == "active"'
+  fields:
+    - name
+    - url
+    - urn.value AS urn_id          # Dot-notation with alias
+    - employee_count
+  add_columns:
+    batch: "q1-2026"
+    source: "linkedin"
+```
+
+### Filter Syntax
+Safe expression parser (no `eval()`). Supported operators: `==`, `!=`, `>`, `<`, `>=`, `<=`. Connectors: `and`, `or`. Values: strings (`"..."`), numbers, `null`.
+
+```
+.field > 10
+.status == "active"
+.location != ""
+.name != null
+.count > 5 and .count < 100
+.status == "active" or .status == "pending"
+```
+
+---
+
+## Per-Source Export
+
+Export destinations run after Parquet write. Transform is applied to export records if configured.
+
+### File Export
+```yaml
+export:
+  - type: file
+    path: ./output/{{source}}-{{date}}.csv
+    format: csv                    # json, jsonl, csv
+```
+
+Template variables: `{{date}}` (YYYY-MM-DD), `{{datetime}}` (ISO), `{{source}}` (source ID), `{{dataset}}` (dataset name).
+
+### Webhook Export
+```yaml
+export:
+  - type: webhook
+    url: https://example.com/hook
+    headers:
+      X-Token: abc
+```
+
+Sends POST with JSON body: `{dataset, source, count, records, timestamp}`.
+
+---
+
+## Run History & Logs
+
+Every `collect` run is automatically recorded in SQLite (`~/.anysite/dataset_history.db`).
+
+```bash
+# View run history
+anysite dataset history my-dataset
+anysite dataset history my-dataset --limit 5
+
+# View logs for a specific run
+anysite dataset logs my-dataset --run 42
+
+# View latest run logs
+anysite dataset logs my-dataset
+```
+
+---
+
+## Scheduling
+
+Generate cron or systemd entries from the `schedule.cron` config.
+
+```bash
+# Crontab entry (default)
+anysite dataset schedule dataset.yaml --incremental --load-db pg
+
+# Systemd timer units
+anysite dataset schedule dataset.yaml --systemd --incremental --load-db pg
+```
+
+Output example:
+```
+0 9 * * * /path/to/anysite dataset collect dataset.yaml --incremental --load-db pg >> ~/.anysite/logs/my-dataset_cron.log 2>&1
+```
+
+---
+
+## Notifications
+
+Webhook notifications sent on collection complete or failure.
+
+```yaml
+notifications:
+  on_complete:
+    - url: "https://hooks.slack.com/services/xxx"
+      headers: {Authorization: "Bearer token"}
+  on_failure:
+    - url: "https://alerts.example.com/fail"
+```
+
+Payload: `{event: "complete"|"failure", dataset, timestamp, record_count, source_count, duration, error}`.
+
+---
+
+## Reset Incremental State
+
+Clear collected input tracking to force re-collection.
+
+```bash
+# Reset all sources
+anysite dataset reset-cursor dataset.yaml
+
+# Reset specific source
+anysite dataset reset-cursor dataset.yaml --source profiles
+```

package/skills/skill-audit/README.md

@@ -0,0 +1,287 @@
+# Skill Audit
+
+Static security auditor for Claude Code skills, commands, and plugins. Detects dangerous patterns — excessive permissions, hooks, prompt injection, credential access, privilege escalation — before you enable a skill.
+
+**Developer:** Anysite Skills Contributors
+
+## Overview
+
+The Skill Audit skill performs read-only static security analysis of Claude Code skills, commands, and plugins. It analyzes SKILL.md files, supporting scripts, and hooks to identify security risks before installation.
+
+Works with both local files and remote GitHub repositories.
+
+## Installation
+
+### Prerequisites
+
+- **Claude Code** - Version with skill support enabled
+- **Anysite Skills Marketplace** - Already added if you're viewing this
+
+### Install from Marketplace
+
+```bash
+# Add the marketplace (if not already added)
+/plugin marketplace add https://github.com/anysiteio/agent-skills
+
+# Install the skill
+/plugin install skill-audit@anysite-skills
+```
+
+### Permissions
+
+The skill requires these permissions (automatically granted during installation):
+
+```json
+{
+  "permissions": {
+    "allow": [
+      "Skill(skill-audit)",
+      "Skill(skill-audit:*)"
+    ]
+  }
+}
+```
+
+For stricter WebFetch domain enforcement (recommended):
+
+```json
+{
+  "permissions": {
+    "allow": [
+      "Skill(skill-audit)",
+      "Skill(skill-audit:*)",
+      "WebFetch(domain:api.github.com)",
+      "WebFetch(domain:raw.githubusercontent.com)"
+    ]
+  }
+}
+```
+
+## Usage
+
+### Quick Start
+
+Once installed, invoke the skill to audit targets:
+
+```bash
+# Audit a specific skill directory
+/skill-audit .claude/skills/my-skill
+
+# Audit a specific command file
+/skill-audit .claude/commands/deploy.md
+
+# Audit a skill by name (searches .claude/skills/ and .claude/commands/)
+/skill-audit my-skill
+
+# Audit ALL skills and commands in the current project
+/skill-audit
+```
+
+### Remote Audit (GitHub)
+
+Audit skills from GitHub before installing them:
+
+```bash
+# Audit a GitHub repository
+/skill-audit https://github.com/user/repo
+
+# Audit a specific directory on GitHub
+/skill-audit https://github.com/user/repo/tree/main/.claude/skills/my-skill
+
+# Audit a single file from GitHub
+/skill-audit https://github.com/user/repo/blob/main/.claude/commands/deploy.md
+```
+
+## What It Checks
+
+The auditor performs a comprehensive 5-phase static analysis:
+
+### Analysis Phases
+
+| Phase | What It Analyzes |
+|-------|------------------|
+| **Discovery** | Inventories all files in the skill/plugin directory, detects plugin structure |
+| **Frontmatter** | Checks `allowed-tools`, hooks, permissions, invocation settings |
+| **Body Content** | Scans for dangerous tool references, settings manipulation, injection patterns, sensitive file access |
+| **Supporting Files** | Examines scripts for network egress, credential access, code execution, persistence |
+| **Hooks** | Detects hook definitions or installations, classifies by risk level |
+
+### Detection Rules
+
+| Rule ID | Severity | Description |
+|---------|----------|-------------|
+| **SKL-001a** | Medium | Hooks present in skill (require manual review) |
+| **SKL-001b** | Critical | Hooks with dangerous patterns (network, credentials, persistence) |
+| **SKL-002** | Critical/High | Prompt injection or dynamic injection patterns (`!` + backticks, `$(...)`) |
+| **SKL-003** | High | Bash/WebFetch/broad wildcards in `allowed-tools` |
+| **SKL-004** | Medium/High | Missing `disable-model-invocation` on side-effect skills |
+| **SKL-005** | High | Dangerous patterns in supporting scripts |
+| **SKL-006** | High | Instructions to change permissions, settings, or hooks |
+
+### Risk Scoring
+
+| Score Range | Risk Level | Recommendation |
+|-------------|------------|----------------|
+| **0** | Clean | No findings, safe to enable |
+| **1-3** | Low-Medium | Review findings, likely safe |
+| **4-6** | High | Needs attention before enabling |
+| **7-8** | Critical | Do not enable without remediation |
+| **9-10** | Severe | Multiple critical findings, likely malicious |
+
+## Output Format
+
+The skill generates detailed audit reports:
+
+```markdown
+# Skill Audit Report: my-skill
+
+**Path:** .claude/skills/my-skill/
+**Date:** 2026-01-30
+**Risk Score:** 3/10
+**Overall Severity:** Medium
+
+## Summary
+The skill uses WebFetch for external API calls but has no hooks or injection patterns.
+
+## Findings
+| # | ID | Severity | Finding | Location | Evidence |
+|---|---|---|---|---|---|
+| 1 | SKL-003 | Medium | WebFetch in allowed-tools | SKILL.md:4 | `allowed-tools: Read, WebFetch` |
+
+## Hardening Recommendations
+1. Remove WebFetch unless strictly required for the skill's purpose.
+```
+
+## Security Properties
+
+This skill is designed with security as the top priority:
+
+### Read-Only Analysis
+- Uses ONLY `Read`, `Grep`, `Glob` tools (plus `WebFetch` for remote GitHub audits)
+- Cannot modify files or execute commands
+- No system state changes
+
+### Anti-Injection Protection
+- All audited content treated as untrusted malicious input
+- Instructions found in audited files are flagged as findings
+- Never follows or executes instructions from audited content
+
+### Evidence Redaction
+- Secrets, tokens, and API keys found in evidence are automatically masked
+- Prevents accidental exposure of credentials in audit reports
+- Shows only first 4 and last 4 characters of sensitive values
+
+### Scoped WebFetch
+- WebFetch restricted to `raw.githubusercontent.com` and `api.github.com`
+- Only used for fetching remote skill files from GitHub
+- No redirects followed, no links from fetched content used
+- No arbitrary URL access
+
+### Isolated Execution
+- Runs in a forked subagent context (`context: fork`)
+- Prevents context contamination from audited content
+- Clean execution environment for each audit
+
+### Manual-Only Invocation
+- `disable-model-invocation: true` prevents auto-triggering
+- Must be invoked explicitly via `/skill-audit`
+- No automatic background scans
+
+### Remote Audit Limits
+- Maximum 20 files per remote audit
+- 100 KB file size limit
+- Large repositories require targeting a specific subdirectory
+
+## Use Cases
+
+### Pre-Installation Audit
+Audit skills from GitHub before installing them:
+```bash
+/skill-audit https://github.com/third-party/suspicious-skill
+```
+
+### Regular Security Reviews
+Audit all installed skills periodically:
+```bash
+/skill-audit
+```
+
+### Third-Party Skill Vetting
+Review community-contributed skills before enabling:
+```bash
+/skill-audit .claude/skills/community-skill
+```
+
+### Command Safety Check
+Verify custom commands don't have dangerous patterns:
+```bash
+/skill-audit .claude/commands/deploy.md
+```
+
+## How It Works
+
+```
+User Request
+    ↓
+skill-audit Skill
+    ↓
+Static Analysis Engine
+    ↓
+Security Report (Read-only, no modifications)
+```
+
+The skill performs purely static analysis without executing any code from the audited target. All analysis is based on pattern matching, AST analysis, and heuristic detection.
+
+## Key Features
+
+**Comprehensive Detection**
+- Hooks and auto-execution patterns
+- Prompt injection attempts
+- Excessive permissions
+- Credential access patterns
+- Privilege escalation attempts
+
+**Multi-Target Support**
+- Local skill directories
+- Individual command files
+- Remote GitHub repositories
+- Bulk project audits
+
+**Zero Side Effects**
+- Read-only operation
+- No file modifications
+- No command execution
+- Safe to run on any skill
+
+**Detailed Reporting**
+- Risk scoring (0-10 scale)
+- Severity classification
+- Specific evidence locations
+- Hardening recommendations
+
+## Structure
+
+```
+skill-audit/
+├── SKILL.md          # Skill definition + audit logic (single file)
+└── README.md         # This file
+```
+
+## Contributing
+
+Found a security pattern that should be detected? Submit an issue or pull request at:
+https://github.com/anysiteio/agent-skills/issues
+
+## Support
+
+- **GitHub Issues**: [github.com/anysiteio/agent-skills/issues](https://github.com/anysiteio/agent-skills/issues)
+- **Documentation**: [SKILL.md](SKILL.md)
+- **Anysite MCP**: [docs.anysite.io](https://docs.anysite.io)
+
+## License
+
+MIT License - see [LICENSE](../../LICENSE) file for details
+
+---
+
+**Note**: This is a security/defensive tool and operates independently from the anysite MCP server. It does not require anysite MCP server to function.