@objctp/opencode-shell-routines 1.2.0

package/skills/shell-profiling/examples/profile-session.md

@@ -0,0 +1,225 @@
+# Example: Profiling a Slow Log-Processing Script
+
+## Scenario
+
+A script that filters log entries by date range, extracts fields, and computes a summary is running slowly on a 500,000-line log file. The script produces correct output but takes over 30 seconds. The goal is to bring it under 5 seconds.
+
+## The Script
+
+```bash
+#!/usr/bin/env bash
+set -euo pipefail
+
+# analyse-logs.sh -- Summarise error counts by service
+# Usage: analyse-logs.sh <start-date> <end-date> <logfile>
+# Dates in YYYY-MM-DD format
+
+start_date="${1:?Usage: $0 <start-date> <end-date> <logfile>}"
+end_date="${2:?Usage: $0 <start-date> <end-date> <logfile>}"
+logfile="${3:?Usage: $0 <start-date> <end-date> <logfile>}"
+
+declare -A service_counts
+total_errors=0
+
+while IFS= read -r line; do
+    # Extract date from line: "2025-01-15 08:23:41 [ERROR] ..."
+    line_date=$(echo "$line" | cut -d' ' -f1)
+
+    # Check date range
+    if [[ "$line_date" < "$start_date" || "$line_date" > "$end_date" ]]; then
+        continue
+    fi
+
+    # Extract severity
+    severity=$(echo "$line" | grep -o '\[[A-Z]\+\]' | tr -d '[]')
+
+    if [[ "$severity" != "ERROR" ]]; then
+        continue
+    fi
+
+    # Extract service name (field after severity)
+    service=$(echo "$line" | cut -d' ' -f4)
+
+    # Count
+    (( service_counts["$service"]++ ))
+    (( total_errors++ ))
+done < "$logfile"
+
+# Print summary
+echo "Error summary: $start_date to $end_date"
+echo "Total errors: $total_errors"
+echo ""
+for service in "${!service_counts[@]}"; do
+    echo "  $service: ${service_counts[$service]}"
+done
+```
+
+## Step 1: Baseline Measurement
+
+Run with `time` to establish the baseline:
+
+```bash
+$ time bash analyse-logs.sh 2025-01-01 2025-01-31 access.log
+
+Error summary: 2025-01-01 to 2025-01-31
+Total errors: 3421
+  auth-service: 1204
+  payment-gateway: 987
+  user-api: 623
+  ...
+
+real    0m32.456s
+user    0m28.123s
+sys     0m4.102s
+```
+
+32 seconds. The high user time relative to wall-clock time suggests CPU-bound work, likely from spawning many external processes.
+
+## Step 2: Xtrace Profiling with Timestamps
+
+Add timestamped tracing to a copy of the script. Insert these lines after `set -euo pipefail`:
+
+```bash
+exec 42>/tmp/analyse-logs.trace.log
+BASH_XTRACEFD=42
+PS4='+ ${EPOCHREALTIME} ${BASH_SOURCE}:${LINENO} ${FUNCNAME[0]:-main} '
+set -x
+```
+
+Run the script again. The trace log now contains timestamped entries:
+
+```bash
+$ head -5 /tmp/analyse-logs.trace.log
++ 1709654321.001234 analyse-logs.sh:12 main
++ 1709654321.001456 analyse-logs.sh:19 main + IFS= read -r line
++ 1709654321.002001 analyse-logs.sh:22 main ++ echo '2025-01-15 08:23:41 [ERROR] auth-service ...'
++ 1709654321.002345 analyse-logs.sh:22 main ++ cut '-d ' -f1
++ 1709654321.003100 analyse-logs.sh:22 main + line_date=2025-01-15
+```
+
+## Step 3: Identify the Hotspot
+
+Post-process the trace to find the slowest lines:
+
+```bash
+scripts/trace-aggregate.sh /tmp/analyse-logs.trace.log 10
+```
+
+Output:
+
+```
+12.3450 s  analyse-logs.sh:22 (called 500000 times)
+ 8.7650 s  analyse-logs.sh:25 (called 350000 times)
+ 5.4320 s  analyse-logs.sh:29 (called 142000 times)
+ 3.2100 s  analyse-logs.sh:33 (called 3421 times)
+ ...
+```
+
+Line 22 (`echo "$line" | cut -d' ' -f1`) is called 500,000 times and accounts for 12 seconds. Line 25 (`grep -o`) accounts for another 8.7 seconds. Together, these subshell invocations in the tight loop dominate execution time.
+
+> In this case, xtrace profiling was sufficient to identify the hotspot. For I/O-bound problems where the bottleneck is file operations or network calls, the deep-dive step (Step 5 in the workflow: `strace -c`, `/usr/bin/time -v`) would be the next step to quantify syscall overhead.
+
+## Step 4: Apply Optimisation
+
+Replace the external commands with parameter expansion and bash builtins. Here is the optimised loop body:
+
+**Before (lines 19--38):**
+
+```bash
+while IFS= read -r line; do
+    line_date=$(echo "$line" | cut -d' ' -f1)
+
+    if [[ "$line_date" < "$start_date" || "$line_date" > "$end_date" ]]; then
+        continue
+    fi
+
+    severity=$(echo "$line" | grep -o '\[[A-Z]\+\]' | tr -d '[]')
+
+    if [[ "$severity" != "ERROR" ]]; then
+        continue
+    fi
+
+    service=$(echo "$line" | cut -d' ' -f4)
+    (( service_counts["$service"]++ ))
+    (( total_errors++ ))
+done < "$logfile"
+```
+
+**After:**
+
+```bash
+while IFS=' ' read -r line_date _time severity_raw service_rest; do
+    # Filter date range
+    if [[ "$line_date" < "$start_date" || "$line_date" > "$end_date" ]]; then
+        continue
+    fi
+
+    # Check severity -- severity_raw looks like "[ERROR]"
+    if [[ "$severity_raw" != "[ERROR]" ]]; then
+        continue
+    fi
+
+    # Extract service name (first word of remaining fields)
+    read -r service _ <<< "$service_rest"
+    (( service_counts["$service"]++ ))
+    (( total_errors++ ))
+done < "$logfile"
+```
+
+**What changed:**
+
+1. `IFS=' ' read -r line_date _time severity_raw service_rest` splits the line into fields in a single `read` call, replacing three `$(echo ... | cut)` and one `$(echo ... | grep | tr)` pipeline -- four subshell invocations per line reduced to zero.
+2. The severity check compares directly against `[ERROR]` instead of extracting and stripping brackets.
+3. The service extraction uses a second `read` with here-string instead of `cut`.
+
+## Step 5: Benchmark Before/After
+
+Using hyperfine:
+
+```bash
+$ hyperfine --warmup 2 --runs 5 \
+    'bash analyse-logs-before.sh 2025-01-01 2025-01-31 access.log' \
+    'bash analyse-logs-after.sh 2025-01-01 2025-01-31 access.log'
+
+Benchmark 1: bash analyse-logs-before.sh ...
+  Time (mean +/- sd):     32.118 s +/-  0.891 s
+
+Benchmark 2: bash analyse-logs-after.sh ...
+  Time (mean +/- sd):      3.452 s +/-  0.134 s
+
+Summary
+  bash analyse-logs-after.sh ran 9.30 +/- 0.48 times faster than bash analyse-logs-before.sh
+```
+
+## Step 6: Summary
+
+| Metric          | Before    | After    | Improvement        |
+| --------------- | --------- | -------- | ------------------ |
+| Wall-clock time | 32.1 s    | 3.5 s    | 9.3x faster        |
+| Subshells/line  | 4         | 0        | eliminated         |
+| External cmds   | cut, grep, tr | read only | builtin only   |
+
+**Pattern applied:** Subshell elimination -- replaced `$(echo ... | cmd)` pipelines with `read` field splitting and parameter expansion.
+
+**Trade-off:** The optimised version requires that log lines follow the expected space-delimited format. If the format changes (e.g., quoted fields containing spaces), the `read`-based splitting would need adjustment. This is acceptable because the original `cut -d' '` approach had the same limitation.
+
+## Further Optimisation (Optional)
+
+For even faster performance on very large files, replace the bash loop entirely with a single `awk` pass:
+
+```bash
+awk -v start="$start_date" -v end="$end_date" '
+    $1 >= start && $1 <= end && $3 == "[ERROR]" {
+        count[$4]++
+        total++
+    }
+    END {
+        printf "Error summary: %s to %s\n", start, end
+        printf "Total errors: %d\n\n", total
+        for (svc in count)
+            printf "  %s: %d\n", svc, count[svc]
+    }
+' "$logfile"
+```
+
+This processes the file in C without returning to the shell between lines. Typical result: under 0.5 seconds for 500,000 lines. The trade-off is moving logic out of bash into awk, which reduces readability for team members unfamiliar with awk.

package/skills/shell-profiling/references/optimisation-patterns.md

@@ -0,0 +1,373 @@
+# Shell Optimisation Patterns
+
+Each pattern shows a "before" (slow) and "after" (fast) version with an explanation.
+
+---
+
+## Subshell Elimination
+
+Every `$(...)` or backtick invocation forks a child process. In a loop of 10,000 iterations, that is 10,000 extra processes.
+
+### Replace external commands with parameter expansion
+
+**Before:**
+
+```bash
+for path in "${paths[@]}"; do
+    base=$(basename "$path")
+    dir=$(dirname "$path")
+    ext="${path##*.}"
+    echo "$dir/$base has extension $ext"
+done
+```
+
+**After:**
+
+```bash
+for path in "${paths[@]}"; do
+    base="${path##*/}"
+    dir="${path%/*}"
+    ext="${path##*.}"
+    echo "$dir/$base has extension $ext"
+done
+```
+
+**Why:** `basename` and `dirname` each fork a process. Parameter expansion `${path##*/}` and `${path%/*}` are bash builtins -- no fork, no exec, no I/O.
+
+### Use `read` with here-string instead of piping to commands
+
+**Before:**
+
+```bash
+echo "$line" | cut -d: -f2
+```
+
+**After:**
+
+```bash
+IFS=: read -r _ field2 _ <<< "$line"
+echo "$field2"
+```
+
+**Why:** The pipe (`|`) creates a subshell for both sides. The here-string (`<<<`) is a builtin that avoids the fork.
+
+### Process substitution vs pipe into loops
+
+**Before:**
+
+```bash
+total=0
+grep 'pattern' data.txt | while IFS= read -r line; do
+    (( total += line ))
+done
+echo "$total"  # 0 -- subshell lost the variable
+```
+
+**After:**
+
+```bash
+total=0
+while IFS= read -r line; do
+    (( total += line ))
+done < <(grep 'pattern' data.txt)
+echo "$total"  # correct value
+```
+
+**Why:** The pipe runs `while` in a subshell, losing variable changes. Process substitution `<(...)` runs `grep` in a subshell instead, keeping the `while` loop in the main shell.
+
+---
+
+## Builtin vs External Commands
+
+### `[[ ]]` vs `[ ]` vs `test`
+
+**Before:**
+
+```bash
+if [ "$status" = "ok" ] && [ "$count" -gt 0 ]; then
+    echo "valid"
+fi
+```
+
+**After:**
+
+```bash
+if [[ "$status" == "ok" && "$count" -gt 0 ]]; then
+    echo "valid"
+fi
+```
+
+**Why:** `[[ ]]` is a bash builtin that supports `&&` and `||` inside the expression, avoiding multiple `[ ]` invocations and the associated parsing overhead. It also handles empty variables without quoting.
+
+### bash string manipulation vs sed/awk
+
+**Before:**
+
+```bash
+lower=$(echo "$name" | tr 'A-Z' 'a-z')
+no_spaces=$(echo "$lower" | sed 's/ /_/g')
+clean=$(echo "$no_spaces" | sed 's/[^a-z0-9_]//g')
+```
+
+**After:**
+
+```bash
+lower="${name,,}"
+no_spaces="${lower// /_}"
+clean="${no_spaces//[^a-z0-9_]/}"
+```
+
+**Why:** Each pipe to `tr` or `sed` forks a process and opens a pipe. The bash parameter expansions `${var,,}`, `${var//pattern/replacement}` are builtins that operate on the string in memory. Three external processes become zero.
+
+### `mapfile`/`readarray` vs `while read` loop
+
+**Before:**
+
+```bash
+lines=()
+while IFS= read -r line; do
+    lines+=("$line")
+done < file.txt
+```
+
+**After:**
+
+```bash
+mapfile -t lines < file.txt
+```
+
+**Why:** `mapfile` reads the entire file into an array in a single builtin operation. The `while read` loop invokes the `read` builtin once per line, with per-iteration overhead.
+
+### Arithmetic `(( ))` vs `expr`
+
+**Before:**
+
+```bash
+result=$(expr "$a" + "$b")
+remainder=$(expr "$a" % "$b")
+```
+
+**After:**
+
+```bash
+(( result = a + b ))
+(( remainder = a % b ))
+```
+
+**Why:** `expr` is an external command that forks a process for each arithmetic operation. `(( ))` is a bash builtin.
+
+---
+
+## I/O Optimisation
+
+### Redirect once vs per-line
+
+**Before:**
+
+```bash
+for item in "${items[@]}"; do
+    echo "$item" >> output.txt
+done
+```
+
+**After:**
+
+```bash
+exec 3>output.txt
+for item in "${items[@]}"; do
+    echo "$item" >&3
+done
+exec 3>&-
+```
+
+**Why:** Each `>>` opens the file, seeks to the end, writes, and closes. Redirecting via `exec 3>` opens the file once; subsequent `>&3` writes skip the open/close cycle.
+
+### Avoid useless `cat`
+
+**Before:**
+
+```bash
+cat file.txt | grep 'pattern'
+cat file.txt | while IFS= read -r line; do ...
+```
+
+**After:**
+
+```bash
+grep 'pattern' file.txt
+while IFS= read -r line; do ... done < file.txt
+```
+
+**Why:** `cat file | cmd` spawns a `cat` process and creates a pipe, both unnecessary when the command can read the file directly via redirection.
+
+### Batch reads vs line-by-line
+
+**Before:**
+
+```bash
+while IFS= read -r line; do
+    echo "$line"
+done < large_file.txt > output.txt
+```
+
+**After:**
+
+```bash
+# If the operation is a simple transformation, use a single tool pass
+sed 's/old/new/g' large_file.txt > output.txt
+
+# Or for more complex processing, use awk
+awk '{gsub(/old/, "new"); print}' large_file.txt > output.txt
+```
+
+**Why:** Reading line-by-line in bash has per-iteration overhead (buffer management, builtin invocation). A single `sed` or `awk` pass processes the entire file in C without returning to the shell between lines.
+
+---
+
+## Loop Optimisation
+
+### Process substitution instead of piping into loops
+
+See **Subshell Elimination** above — `command | while ...` loses variables; use `while ... done < <(command)`.
+
+### `shopt -s lastpipe`
+
+**Before:**
+
+```bash
+# Variables set in the last command of a pipe are lost
+printf '%s\n' "${items[@]}" | sort | while IFS= read -r item; do
+    (( count++ ))
+done
+echo "$count"  # empty
+```
+
+**After:**
+
+```bash
+shopt -s lastpipe
+count=0
+printf '%s\n' "${items[@]}" | sort | while IFS= read -r item; do
+    (( count++ ))
+done
+echo "$count"  # correct value
+```
+
+**Why:** `lastpipe` runs the last command of a pipeline in the current shell (not a subshell), preserving variable assignments. Requires job control to be disabled (which it is in scripts by default).
+
+### `xargs -P` for parallel work
+
+**Before:**
+
+```bash
+for url in "${urls[@]}"; do
+    curl -s "$url" > /dev/null
+done
+```
+
+**After:**
+
+```bash
+printf '%s\n' "${urls[@]}" | xargs -P 8 -I{} curl -s "{}" > /dev/null
+```
+
+**Why:** The sequential loop processes one URL at a time. `xargs -P 8` runs up to 8 curl processes in parallel, bounded to avoid overwhelming the system.
+
+---
+
+## String Processing
+
+### Parameter expansion vs external tools for simple transformations
+
+**Before:**
+
+```bash
+trimmed=$(echo "$var" | sed 's/^[[:space:]]*//;s/[[:space:]]*$//')
+```
+
+**After:**
+
+```bash
+# Bash 4.4+ does not trim by default, so read into a temporary
+read -r trimmed <<< "$var"
+```
+
+**Why:** The `sed` pipeline forks two processes. `read` with here-string is a single builtin.
+
+### Single-pass awk vs multi-tool pipeline
+
+**Before:**
+
+```bash
+grep 'ERROR' log.txt | cut -d' ' -f3 | sort | uniq -c | sort -rn | head -10
+```
+
+**After:**
+
+```bash
+awk '/ERROR/ {count[$3]++} END {for (k in count) print count[k], k}' log.txt \
+    | sort -rn | head -10
+```
+
+**Why:** The before version pipes through 5 processes (grep, cut, sort, uniq, sort). The after version does filtering, field extraction, and counting in a single `awk` pass, reducing the pipeline to 2 processes (awk + sort).
+
+### Replace entire bash loop with a single awk pass
+
+For data-heavy scripts, the single highest-impact optimisation is often replacing the entire bash loop with awk.
+
+**Before:**
+
+```bash
+declare -A counts
+total=0
+while IFS= read -r line; do
+    date=$(echo "$line" | cut -d' ' -f1)
+    if [[ "$date" < "$start" || "$date" > "$end" ]]; then continue; fi
+    severity=$(echo "$line" | grep -o '\[[A-Z]\+\]' | tr -d '[]')
+    if [[ "$severity" != "ERROR" ]]; then continue; fi
+    service=$(echo "$line" | cut -d' ' -f4)
+    (( counts["$service"]++ ))
+    (( total++ ))
+done < "$logfile"
+```
+
+**After:**
+
+```bash
+awk -v start="$start" -v end="$end" '
+    $1 >= start && $1 <= end && $3 == "[ERROR]" {
+        count[$4]++
+        total++
+    }
+    END {
+        printf "Total errors: %d\n", total
+        for (svc in count)
+            printf "  %s: %d\n", svc, count[svc]
+    }
+' "$logfile"
+```
+
+**Why:** The bash version spawns 3-4 external processes per line (echo, cut, grep, tr) across potentially hundreds of thousands of iterations. The awk version processes the entire file in C without returning to the shell between lines, typically achieving 50-100x speedups on large files. The trade-off is moving logic out of bash into awk, which reduces readability for team members unfamiliar with awk.
+
+### `printf %s` concatenation vs repeated echo
+
+**Before:**
+
+```bash
+echo "Header" > output.txt
+echo "$line1" >> output.txt
+echo "$line2" >> output.txt
+echo "Footer" >> output.txt
+```
+
+**After:**
+
+```bash
+{
+    printf '%s\n' "Header"
+    printf '%s\n' "$line1" "$line2"
+    printf '%s\n' "Footer"
+} > output.txt
+```
+
+**Why:** Grouping with `{ }` redirects once for the entire block. `printf '%s\n'` can print multiple arguments in a single call, reducing the number of builtin invocations. Using `printf` instead of `echo` also avoids portability issues with escape sequences.