@hybridlabor-api/aos 4.8.0 → 4.9.0

package/skills/global_config/bash-script-validator/SKILL.md

@@ -0,0 +1,249 @@
+---
+name: bash-script-validator
+description: Validate, lint, audit, or fix bash/shell/.sh scripts via ShellCheck.
+category: engineering-method
+source: cc-devops-skills
+date_added: "2026-09-25"
+---
+
+# Bash Script Validator
+
+## Overview
+
+This skill validates Bash and POSIX shell scripts with layered checks:
+
+1. Syntax validation (`bash -n` or `sh -n`)
+2. ShellCheck static analysis (system binary or wrapper fallback)
+3. Custom security, portability, and optimization checks
+
+Use the default flow below, then branch to fallbacks only when the environment is constrained.
+
+## Trigger Guidance
+
+Use this skill when the request includes script quality, linting, syntax checking, or shell portability work.
+
+### Trigger Phrases
+
+- "Validate this bash script"
+- "Lint this `.sh` file"
+- "Find security issues in this shell script"
+- "Why does this script fail ShellCheck?"
+- "Make this script POSIX compliant"
+- "Review this shell script before CI"
+
+### Non-Trigger Examples
+
+- General Linux command questions with no script file
+- Kubernetes, Terraform, or pipeline validation tasks that do not involve shell scripts
+- Pure prose editing tasks
+
+## Deterministic Execution Model
+
+Run commands from this skill directory:
+
+```bash
+cd devops-skills-plugin/skills/bash-script-validator
+```
+
+### Step 1: Preflight
+
+1. Confirm target path exists and is readable.
+2. Confirm `bash` is available.
+3. Determine whether fixes can be applied directly (write access) or only suggested (read-only).
+
+### Step 2: Run Baseline Validation (Default Path)
+
+```bash
+bash scripts/validate.sh <script-path>
+```
+
+For deterministic stage behavior, set the ShellCheck provider explicitly:
+
+```bash
+# Modes: auto (default), system, wrapper, disabled
+VALIDATOR_SHELLCHECK_MODE=system bash scripts/validate.sh <script-path>
+```
+
+Record:
+
+- Detected shell type
+- Exit code (`0` clean, `1` warnings, `2` errors)
+- All reported issue lines and ShellCheck codes (`SC####`) when present
+
+### Step 3: Load Only Needed References
+
+Progressive disclosure by issue type:
+
+- ShellCheck code explanations: `docs/shellcheck-reference.md`
+- General fix patterns and security mistakes: `docs/common-mistakes.md`
+- Bash-only behavior: `docs/bash-reference.md`
+- POSIX portability or bashism fixes: `docs/shell-reference.md`
+- Text-processing optimization issues: `docs/grep-reference.md`, `docs/awk-reference.md`, `docs/sed-reference.md`, `docs/regex-reference.md` (only when directly relevant)
+
+### Step 4: Provide or Apply Fixes
+
+For each issue, include:
+
+1. Exact location from validator output (line number and snippet)
+2. Root cause
+3. Corrected code
+4. Why the change is safer or more portable
+5. Subsection-level citation (format below)
+
+If the request includes patching files and write access is available, apply fixes in small batches grouped by issue type.
+
+### Step 5: Rerun Policy (Mandatory After Changes)
+
+After each batch of edits, rerun the validator:
+
+```bash
+bash scripts/validate.sh <script-path>
+```
+
+Rerun loop rules:
+
+1. Continue until no new errors are introduced.
+2. If warnings remain by design, document why they are intentionally accepted.
+3. If constraints prevent full resolution, report unresolved items with a clear next action.
+4. Always report the latest rerun exit code and remaining issue count.
+
+## Fallback Behavior
+
+Use these branches only when the default flow cannot run as-is.
+
+| Constraint | Fallback action | Reporting requirement |
+| --- | --- | --- |
+| `shellcheck` missing, wrapper available | Let `scripts/validate.sh` use `scripts/shellcheck_wrapper.sh --cache` automatically | State that wrapper mode was used |
+| `shellcheck` and wrapper unavailable | Run syntax + custom checks only (validator does this) | Explicitly call out reduced coverage and missing ShellCheck analysis |
+| Python unavailable for wrapper | Skip wrapper path, keep syntax + custom checks | State why ShellCheck could not run |
+| Target file is read-only | Provide precise patch suggestions without editing | Mark response as "advisory only" |
+| Target file missing or unreadable | Stop and request a valid file path | Do not fabricate results |
+| Binary/non-text input | Stop validation | Report unsupported input type |
+
+## Citation Guidance for Fixes
+
+Use subsection-level citations for every non-trivial fix.
+
+Required citation format:
+
+```text
+Reference: docs/<file>.md -> <Section> -> <Subsection>
+```
+
+Examples:
+
+- `Reference: docs/common-mistakes.md -> 1. Unquoted Variables -> Solution`
+- `Reference: docs/shellcheck-reference.md -> SC2164: Use || exit After cd`
+- `Reference: docs/shell-reference.md -> POSIX Best Practices -> 5. Avoid Bashisms`
+
+Citation rules:
+
+1. Cite the most specific section that justifies the fix.
+2. For ShellCheck findings, include both the `SC####` code and the matching section.
+3. If no exact subsection exists, cite the closest section and state that the fix is inferred from that guidance.
+
+## Response Template
+
+Use this structure for deterministic output:
+
+- `Validation Results`
+- `Command`: `bash scripts/validate.sh <script-path>`
+- `Detected shell`: `<shell>`
+- `Exit code`: `<code>`
+- `Summary`: `<errors> errors, <warnings> warnings, <info> info`
+- `Issue`: `<short label> (Line <n>)`
+- `Problem`:
+  ```bash
+  <problematic snippet>
+  ```
+- `Fix`:
+  ```bash
+  <corrected snippet>
+  ```
+- `Why`: `<short explanation>`
+- `Reference`: `docs/<file>.md -> <Section> -> <Subsection>`
+- `Rerun command`: `bash scripts/validate.sh <script-path>`
+- `Exit code after fixes`: `<code>`
+- `Remaining issues`: `<count or none>`
+
+## Example Flows
+
+### Fully Automated Environment
+
+```bash
+# 1) Baseline validation
+bash scripts/validate.sh examples/bad-bash.sh
+
+# 2) Apply fixes to target script
+# 3) Rerun validation
+bash scripts/validate.sh examples/bad-bash.sh
+```
+
+Expected behavior: full syntax + ShellCheck + custom-check coverage, with iterative reruns until stable.
+
+### Deterministic CI Gate
+
+```bash
+# Requires a system shellcheck binary.
+bash scripts/run_ci_checks.sh
+```
+
+This runner enforces `VALIDATOR_REQUIRE_SHELLCHECK=1` and `VALIDATOR_SHELLCHECK_MODE=system`
+so CI fails if the ShellCheck stage is skipped or unavailable.
+
+### Constrained Environment (No ShellCheck Runtime)
+
+```bash
+# shellcheck unavailable and wrapper cannot run
+bash scripts/validate.sh examples/bad-shell.sh
+```
+
+Expected behavior: syntax + custom checks still run. Report reduced coverage and list what must be revalidated once ShellCheck is available.
+
+## Validator Script Details
+
+### Scripts
+
+- `scripts/validate.sh`: primary validator entrypoint
+- `scripts/shellcheck_wrapper.sh`: optional ShellCheck fallback using a cached Python virtual environment
+
+### Detection and Ordering
+
+Validation order in `scripts/validate.sh`:
+
+1. File checks (exists/readable/text)
+2. Shebang-based shell detection
+3. Syntax check
+4. ShellCheck (or fallback/skip behavior)
+5. Custom checks
+6. Summary with exit code
+
+### Exit Codes
+
+- `0`: no issues found
+- `1`: warnings found
+- `2`: errors found
+
+## References
+
+Load only what is needed:
+
+- `docs/bash-reference.md`
+- `docs/shell-reference.md`
+- `docs/shellcheck-reference.md`
+- `docs/common-mistakes.md`
+- `docs/grep-reference.md`
+- `docs/awk-reference.md`
+- `docs/sed-reference.md`
+- `docs/regex-reference.md`
+
+## Done Criteria
+
+This skill update is complete when all are true:
+
+1. Trigger guidance is explicit (positive and non-trigger examples).
+2. Default workflow is deterministic and ordered.
+3. Fallback behavior is explicit for missing tooling and constrained environments.
+4. Fix explanations include subsection-level citations.
+5. Post-fix rerun policy is mandatory and reported with exit codes.
+6. Documentation supports both fully automated and constrained execution paths.

package/skills/global_config/bash-script-validator/docs/awk-reference.md

@@ -0,0 +1,449 @@
+# GNU AWK (gawk) Reference Guide
+
+## Overview
+
+AWK is a powerful text processing language designed for pattern scanning and processing. It's particularly useful for field-based data manipulation.
+
+**Official Manual:** https://www.gnu.org/software/gawk/manual/
+**Man Page:** `man awk`
+
+## Basic Syntax
+
+```bash
+awk 'pattern { action }' file
+awk -F delimiter 'pattern { action }' file
+awk -f script.awk file
+```
+
+## Structure
+
+```awk
+BEGIN { # Executed before processing }
+pattern { # Executed for matching lines }
+END { # Executed after processing }
+```
+
+## Built-in Variables
+
+### Field Variables
+```awk
+$0      # Entire line
+$1      # First field
+$2      # Second field
+$NF     # Last field
+$(NF-1) # Second to last field
+```
+
+### Control Variables
+```awk
+NF      # Number of fields in current record
+NR      # Current record number (line number)
+FNR     # Record number in current file
+FS      # Input field separator (default: whitespace)
+OFS     # Output field separator (default: space)
+RS      # Input record separator (default: newline)
+ORS     # Output record separator (default: newline)
+FILENAME # Current filename
+```
+
+## Common Usage Patterns
+
+### Basic Field Processing
+```bash
+# Print specific fields
+awk '{print $1}' file.txt                # First field
+awk '{print $1, $3}' file.txt            # First and third fields
+awk '{print $NF}' file.txt               # Last field
+
+# Print with custom separator
+awk '{print $1 ":" $2}' file.txt         # Custom separator
+awk -v OFS='\t' '{print $1, $2}' file    # Tab-separated
+
+# Print entire line with line number
+awk '{print NR, $0}' file.txt
+```
+
+### Custom Field Separator
+```bash
+# Use comma as separator
+awk -F',' '{print $1}' file.csv
+
+# Use colon as separator (like /etc/passwd)
+awk -F':' '{print $1}' /etc/passwd
+
+# Multiple character separator
+awk -F'::' '{print $2}' file.txt
+
+# Regex as separator
+awk -F'[,:]' '{print $1}' file.txt       # Comma or colon
+```
+
+### Pattern Matching
+```bash
+# Match lines containing pattern
+awk '/pattern/ {print}' file.txt
+awk '/error/ {print $0}' logfile.txt
+
+# Case-insensitive match
+awk 'tolower($0) ~ /pattern/' file.txt
+
+# Regex on specific field
+awk '$2 ~ /pattern/' file.txt
+awk '$2 !~ /pattern/' file.txt  # Not matching
+
+# Exact match
+awk '$1 == "value"' file.txt
+awk '$2 != "value"' file.txt
+```
+
+### Numeric Comparisons
+```bash
+# Greater than
+awk '$3 > 100' file.txt
+
+# Less than or equal
+awk '$2 <= 50' file.txt
+
+# Range
+awk '$1 >= 10 && $1 <= 20' file.txt
+
+# Complex conditions
+awk '$1 > 100 && $2 == "active"' file.txt
+awk '$1 > 100 || $2 > 200' file.txt
+```
+
+### BEGIN and END Blocks
+```bash
+# Header and footer
+awk 'BEGIN {print "Name\tAge"} {print $1, $2} END {print "---"}' file
+
+# Initialize variables
+awk 'BEGIN {count=0} {count++} END {print count}' file
+
+# Set field separator in BEGIN
+awk 'BEGIN {FS=","} {print $1}' file.csv
+```
+
+### Calculations
+```bash
+# Sum a column
+awk '{sum += $1} END {print sum}' file.txt
+
+# Average
+awk '{sum += $1; count++} END {print sum/count}' file
+
+# Count records
+awk 'END {print NR}' file.txt
+
+# Max value
+awk 'BEGIN {max=0} {if ($1 > max) max=$1} END {print max}' file
+
+# Count occurrences
+awk '{count[$1]++} END {for (key in count) print key, count[key]}' file
+```
+
+### Conditional Processing
+```bash
+# If-else
+awk '{if ($1 > 100) print "High"; else print "Low"}' file
+
+# Ternary operator
+awk '{print ($1 > 100) ? "High" : "Low"}' file
+
+# Multiple conditions
+awk '{
+    if ($1 > 100) print "High"
+    else if ($1 > 50) print "Medium"
+    else print "Low"
+}' file
+```
+
+## Arrays
+
+### Associative Arrays
+```awk
+# Count occurrences
+awk '{count[$1]++} END {for (key in count) print key, count[key]}' file
+
+# Group by key
+awk '{sum[$1] += $2} END {for (key in sum) print key, sum[key]}' file
+
+# Check if key exists
+awk '{if ($1 in array) print "Duplicate"}' file
+```
+
+### Array Examples
+```bash
+# Count unique values
+awk '{a[$1]++} END {print length(a)}' file
+
+# Find duplicates
+awk '{count[$1]++} END {for (k in count) if (count[k] > 1) print k}' file
+
+# Store and print in order
+awk '{lines[NR] = $0} END {for (i=1; i<=NR; i++) print lines[i]}' file
+```
+
+## Functions
+
+### Built-in String Functions
+```awk
+length(string)           # String length
+substr(string, start, len) # Substring
+index(string, substring) # Find substring position
+split(string, array, sep) # Split string into array
+sub(regex, replacement, string) # Replace first match
+gsub(regex, replacement, string) # Replace all matches
+tolower(string)          # Convert to lowercase
+toupper(string)          # Convert to uppercase
+match(string, regex)     # Test regex match
+```
+
+### String Function Examples
+```bash
+# String length
+awk '{print length($1)}' file
+
+# Substring
+awk '{print substr($1, 1, 3)}' file  # First 3 characters
+
+# Replace
+awk '{gsub(/old/, "new"); print}' file
+
+# Convert case
+awk '{print toupper($1)}' file
+
+# Split and process
+awk '{split($0, a, ":"); print a[1]}' /etc/passwd
+```
+
+### Math Functions
+```awk
+int(x)      # Integer part
+sqrt(x)     # Square root
+sin(x)      # Sine
+cos(x)      # Cosine
+atan2(y,x)  # Arctangent
+log(x)      # Natural logarithm
+exp(x)      # Exponential
+rand()      # Random number [0,1)
+srand()     # Seed random number generator
+```
+
+## Practical Examples for Shell Scripts
+
+### Log File Analysis
+```bash
+# Count HTTP status codes
+awk '{print $9}' access.log | sort | uniq -c
+
+# Sum response times
+awk '{sum += $10; count++} END {print sum/count}' access.log
+
+# Filter by time range
+awk '$4 > "[01/Jan/2025:10:00:00" && $4 < "[01/Jan/2025:11:00:00"' access.log
+
+# Extract and count IPs
+awk '{print $1}' access.log | sort | uniq -c | sort -rn | head -10
+```
+
+### CSV Processing
+```bash
+# Print specific columns from CSV
+awk -F',' '{print $1, $3}' file.csv
+
+# Skip header
+awk -F',' 'NR > 1 {print $1, $2}' file.csv
+
+# Convert CSV to tab-separated
+awk -F',' -v OFS='\t' '{print $1, $2, $3}' file.csv
+
+# Filter rows
+awk -F',' '$3 > 100 {print $0}' file.csv
+```
+
+### System Administration
+```bash
+# Parse /etc/passwd
+awk -F':' '{print $1, $6}' /etc/passwd  # username, home directory
+awk -F':' '$3 >= 1000 {print $1}' /etc/passwd  # Regular users
+
+# Disk usage analysis
+df -h | awk '$5 > 80 {print $0}'  # > 80% full
+
+# Process monitoring
+ps aux | awk '$3 > 50 {print $2, $11}'  # High CPU processes
+
+# Network stats
+netstat -an | awk '/ESTABLISHED/ {print $5}' | cut -d: -f1 | sort | uniq -c
+```
+
+### Data Transformation
+```bash
+# Swap columns
+awk '{print $2, $1}' file
+
+# Add line numbers
+awk '{print NR ":", $0}' file
+
+# Remove duplicates (keeping first occurrence)
+awk '!seen[$0]++' file
+
+# Join lines
+awk '{printf "%s ", $0} END {print ""}' file
+
+# Transpose rows to columns
+awk '{for (i=1; i<=NF; i++) a[NR,i]=$i; max=(NF>max?NF:max)}
+     END {for (i=1; i<=max; i++) {
+         for (j=1; j<=NR; j++) printf "%s ", a[j,i]
+         print ""
+     }}' file
+```
+
+## Multi-line AWK Scripts
+
+### In Shell Script
+```bash
+awk '
+BEGIN {
+    FS = ","
+    print "Processing..."
+}
+{
+    sum += $2
+    count++
+}
+END {
+    print "Average:", sum/count
+}
+' file.csv
+```
+
+### External AWK File
+```bash
+# script.awk
+BEGIN {
+    FS = ","
+}
+{
+    sum += $2
+}
+END {
+    print "Total:", sum
+}
+
+# Execute
+awk -f script.awk file.csv
+```
+
+## Common Patterns
+
+### Skip Empty Lines
+```bash
+awk 'NF > 0' file
+awk '/./' file
+```
+
+### Print Line Range
+```bash
+awk 'NR>=10 && NR<=20' file  # Lines 10-20
+```
+
+### Print Unique Lines
+```bash
+awk '!seen[$0]++' file
+```
+
+### Count Pattern Occurrences
+```bash
+awk '/pattern/ {count++} END {print count}' file
+```
+
+### Format Output
+```bash
+# Fixed-width columns
+awk '{printf "%-20s %10s\n", $1, $2}' file
+
+# Align numbers
+awk '{printf "%s: %8.2f\n", $1, $2}' file
+```
+
+## Performance Tips
+
+### 1. Use Built-in Variables
+```bash
+# Faster
+awk '{print $NF}' file
+
+# Slower
+awk '{print $length($0)}' file
+```
+
+### 2. Avoid Unnecessary Operations
+```bash
+# Good
+awk '$1 > 100' file
+
+# Wasteful
+awk '{if ($1 > 100) print $0}' file
+```
+
+### 3. Use Regex Efficiently
+```bash
+# Compile regex once
+awk 'BEGIN {pattern = /error/} $0 ~ pattern' file
+```
+
+## Common Pitfalls in Shell Scripts
+
+### 1. Not Quoting AWK Scripts
+```bash
+# Wrong - shell expands $1
+awk {print $1} file
+
+# Right
+awk '{print $1}' file
+```
+
+### 2. Division by Zero
+```bash
+# Dangerous
+awk '{print $1/$2}' file
+
+# Safe
+awk '{if ($2 != 0) print $1/$2; else print "N/A"}' file
+```
+
+### 3. Floating Point Comparison
+```bash
+# Problematic
+awk '$1 == 0.1' file
+
+# Better
+awk 'function abs(x){return x<0?-x:x} abs($1 - 0.1) < 0.001' file
+```
+
+### 4. Not Handling Missing Fields
+```bash
+# Check field existence
+awk 'NF >= 3 {print $3}' file
+```
+
+## Combining with Other Tools
+
+```bash
+# awk with grep
+grep "error" log | awk '{print $1, $NF}'
+
+# awk with sort
+awk '{print $2}' file | sort -n
+
+# Pipeline
+cat file | awk '$1 > 100' | sort | uniq
+```
+
+## Resources
+
+- [GNU AWK Manual](https://www.gnu.org/software/gawk/manual/)
+- [AWK Programming Language](https://www.amazon.com/AWK-Programming-Language-Alfred-Aho/dp/020107981X)
+- [gawk(1) Man Page](https://man7.org/linux/man-pages/man1/gawk.1.html)