command-stream 0.9.2 → 0.9.4

package/README.md

@@ -1334,6 +1334,69 @@ const quickResult = $`pwd`.sync();
 $`npm install`.on('stdout', showProgress).start();
 ```
 
+## Common Pitfalls
+
+### Array Argument Handling
+
+When passing multiple arguments, pass the array directly - **never use `.join(' ')`** before interpolation:
+
+```javascript
+import { $ } from 'command-stream';
+
+// WRONG - entire string becomes ONE argument
+const args = ['file.txt', '--public', '--verbose'];
+await $`command ${args.join(' ')}`;
+// Shell receives: command 'file.txt --public --verbose' (1 argument!)
+// Error: File does not exist: "file.txt --public --verbose"
+
+// CORRECT - each element becomes separate argument
+await $`command ${args}`;
+// Shell receives: command file.txt --public --verbose (3 arguments)
+```
+
+This is a common mistake that causes errors like:
+
+```
+Error: File does not exist: "/path/to/file.txt --flag --option"
+```
+
+### Why This Happens
+
+The `$` template tag handles arrays specially - each element is quoted separately:
+
+```javascript
+if (Array.isArray(value)) {
+  return value.map(quote).join(' '); // Each element quoted individually
+}
+```
+
+But when you call `.join(' ')` first:
+
+1. The array becomes a string: `"file.txt --public --verbose"`
+2. Template receives a **string**, not an array
+3. The entire string gets quoted as one argument
+4. Command receives one argument containing spaces
+
+### Recommended Patterns
+
+```javascript
+// Pattern 1: Direct array passing
+const args = ['file.txt', '--verbose'];
+await $`command ${args}`;
+
+// Pattern 2: Separate interpolations
+const file = 'file.txt';
+const flags = ['--verbose', '--force'];
+await $`command ${file} ${flags}`;
+
+// Pattern 3: Build array dynamically
+const allArgs = ['input.txt'];
+if (verbose) allArgs.push('--verbose');
+await $`command ${allArgs}`;
+```
+
+See [js/BEST-PRACTICES.md](js/BEST-PRACTICES.md) for more detailed guidance.
+
 ## Testing
 
 ```bash

package/js/BEST-PRACTICES.md

@@ -0,0 +1,376 @@
+# Best Practices for command-stream
+
+This document covers best practices, common patterns, and pitfalls to avoid when using the command-stream library.
+
+## Table of Contents
+
+- [Array Argument Handling](#array-argument-handling)
+- [String Interpolation](#string-interpolation)
+- [Security Best Practices](#security-best-practices)
+- [Error Handling](#error-handling)
+- [Performance Tips](#performance-tips)
+- [Common Pitfalls](#common-pitfalls)
+
+---
+
+## Array Argument Handling
+
+### Pass Arrays Directly
+
+When you have multiple arguments in an array, pass the array directly to template interpolation. The library will automatically handle proper quoting for each element.
+
+```javascript
+import { $ } from 'command-stream';
+
+// CORRECT: Pass array directly
+const args = ['file.txt', '--public', '--verbose'];
+await $`command ${args}`;
+// Executed: command file.txt --public --verbose
+
+// CORRECT: Dynamic array building
+const baseArgs = ['input.txt'];
+if (isVerbose) baseArgs.push('--verbose');
+if (isForce) baseArgs.push('--force');
+await $`mycommand ${baseArgs}`;
+```
+
+### Never Use .join() Before Interpolation
+
+Calling `.join(' ')` on an array before passing to template interpolation is a common mistake that causes all elements to become a single argument.
+
+```javascript
+// WRONG: Array becomes single argument
+const args = ['file.txt', '--flag'];
+await $`command ${args.join(' ')}`;
+// Shell receives: ['command', 'file.txt --flag'] (1 argument!)
+
+// CORRECT: Each element becomes separate argument
+await $`command ${args}`;
+// Shell receives: ['command', 'file.txt', '--flag'] (2 arguments)
+```
+
+### Mixed Static and Dynamic Arguments
+
+When combining static and dynamic arguments, use separate interpolations or arrays:
+
+```javascript
+// CORRECT: Multiple interpolations
+const file = 'data.txt';
+const flags = ['--verbose', '--force'];
+await $`process ${file} ${flags}`;
+
+// CORRECT: Build complete array
+const allArgs = [file, ...flags];
+await $`process ${allArgs}`;
+
+// WRONG: String concatenation
+await $`process ${file + ' ' + flags.join(' ')}`;
+```
+
+---
+
+## String Interpolation
+
+### Safe Interpolation (Default)
+
+By default, all interpolated values are automatically quoted to prevent shell injection:
+
+```javascript
+// User input is safely escaped
+const userInput = "'; rm -rf /; echo '";
+await $`echo ${userInput}`;
+// Executed safely - input is quoted, not executed
+```
+
+### Using raw() for Trusted Commands
+
+Only use `raw()` with trusted, hardcoded command strings:
+
+```javascript
+import { $, raw } from 'command-stream';
+
+// CORRECT: Trusted command template
+const trustedCmd = 'git log --oneline --graph';
+await $`${raw(trustedCmd)}`;
+
+// WRONG: User input with raw (security vulnerability!)
+const userInput = req.body.command;
+await $`${raw(userInput)}`; // DANGER: Shell injection!
+```
+
+### Paths with Spaces
+
+Paths containing spaces are automatically quoted:
+
+```javascript
+const path = '/Users/name/My Documents/file.txt';
+await $`cat ${path}`;
+// Executed: cat '/Users/name/My Documents/file.txt'
+```
+
+---
+
+## Security Best Practices
+
+### Never Trust User Input
+
+Always treat external input as potentially malicious:
+
+```javascript
+// CORRECT: Auto-escaping protects against injection
+const filename = req.query.file;
+await $`cat ${filename}`;
+
+// WRONG: Bypassing safety for user input
+await $`${raw(userInput)}`;
+```
+
+### Validate Before Execution
+
+Add validation for critical operations:
+
+```javascript
+import { $ } from 'command-stream';
+
+async function deleteFile(filename) {
+  // Validate filename
+  if (filename.includes('..') || filename.startsWith('/')) {
+    throw new Error('Invalid filename');
+  }
+
+  await $`rm ${filename}`;
+}
+```
+
+### Use Principle of Least Privilege
+
+Run commands with minimal required permissions:
+
+```javascript
+// Use specific paths instead of wildcards when possible
+await $`rm ${specificFile}`; // Better
+await $`rm ${directory}/*`; // More risky
+```
+
+---
+
+## Error Handling
+
+### Check Exit Codes
+
+By default, commands don't throw on non-zero exit codes:
+
+```javascript
+const result = await $`ls nonexistent`;
+if (result.code !== 0) {
+  console.error('Command failed:', result.stderr);
+}
+```
+
+### Enable errexit for Critical Operations
+
+Use shell settings for scripts that should fail on errors:
+
+```javascript
+import { $, shell } from 'command-stream';
+
+shell.errexit(true);
+
+try {
+  await $`critical-operation`;
+} catch (error) {
+  console.error('Critical operation failed:', error);
+  process.exit(1);
+}
+```
+
+### Handle Specific Errors
+
+```javascript
+const result = await $`command`;
+
+switch (result.code) {
+  case 0:
+    console.log('Success:', result.stdout);
+    break;
+  case 1:
+    console.error('General error');
+    break;
+  case 127:
+    console.error('Command not found');
+    break;
+  default:
+    console.error(`Unknown error (code ${result.code})`);
+}
+```
+
+---
+
+## Performance Tips
+
+### Use Streaming for Large Outputs
+
+For commands that produce large outputs, use streaming to avoid memory issues:
+
+```javascript
+// Memory efficient: Process chunks as they arrive
+for await (const chunk of $`cat huge-file.log`.stream()) {
+  processChunk(chunk.data);
+}
+
+// Memory intensive: Buffers entire output
+const result = await $`cat huge-file.log`;
+processAll(result.stdout);
+```
+
+### Parallel Execution
+
+Run independent commands in parallel:
+
+```javascript
+// Sequential (slower)
+await $`task1`;
+await $`task2`;
+await $`task3`;
+
+// Parallel (faster)
+await Promise.all([$`task1`, $`task2`, $`task3`]);
+```
+
+### Use Built-in Commands
+
+Built-in commands are faster as they don't spawn system processes:
+
+```javascript
+// Fast: Built-in command (pure JavaScript)
+await $`mkdir -p build/output`;
+
+// Slower: System command
+await $`/bin/mkdir -p build/output`;
+```
+
+---
+
+## Common Pitfalls
+
+### 1. Array.join() Pitfall (Most Common)
+
+**Problem:** Using `.join(' ')` before interpolation merges all arguments into one.
+
+```javascript
+// WRONG
+const args = ['file.txt', '--flag'];
+await $`cmd ${args.join(' ')}`; // 1 argument: "file.txt --flag"
+
+// CORRECT
+await $`cmd ${args}`; // 2 arguments: "file.txt", "--flag"
+```
+
+See [Case Study: Issue #153](./docs/case-studies/issue-153/README.md) for detailed analysis.
+
+### 2. Template String Concatenation
+
+**Problem:** Building commands with template strings creates single arguments.
+
+```javascript
+// WRONG
+const file = 'data.txt';
+const flag = '--verbose';
+await $`cmd ${`${file} ${flag}`}`; // 1 argument: "data.txt --verbose"
+
+// CORRECT
+await $`cmd ${file} ${flag}`; // 2 arguments
+```
+
+### 3. Forgetting await
+
+**Problem:** Commands return promises, forgetting await causes issues.
+
+```javascript
+// WRONG: Command may not complete before next line
+$`setup-task`;
+$`main-task`; // May run before setup completes
+
+// CORRECT: Wait for completion
+await $`setup-task`;
+await $`main-task`;
+```
+
+### 4. Assuming Synchronous Behavior
+
+**Problem:** Expecting immediate results without awaiting.
+
+```javascript
+// WRONG
+const cmd = $`echo hello`;
+console.log(cmd.stdout); // undefined - not yet executed!
+
+// CORRECT
+const result = await $`echo hello`;
+console.log(result.stdout); // "hello\n"
+```
+
+### 5. Not Handling stderr
+
+**Problem:** Only checking stdout when errors go to stderr.
+
+```javascript
+// INCOMPLETE
+const result = await $`command`;
+console.log(result.stdout);
+
+// BETTER
+const result = await $`command`;
+if (result.code !== 0) {
+  console.error('Error:', result.stderr);
+} else {
+  console.log('Success:', result.stdout);
+}
+```
+
+### 6. Ignoring Exit Codes
+
+**Problem:** Assuming success without checking.
+
+```javascript
+// WRONG
+const result = await $`risky-command`;
+processOutput(result.stdout); // May be empty on failure!
+
+// CORRECT
+const result = await $`risky-command`;
+if (result.code === 0) {
+  processOutput(result.stdout);
+} else {
+  handleError(result);
+}
+```
+
+---
+
+## Quick Reference
+
+### Do's
+
+- Pass arrays directly: `${args}`
+- Use separate interpolations: `${file} ${flag}`
+- Check exit codes after execution
+- Use streaming for large outputs
+- Validate user input before execution
+- Use built-in commands when available
+
+### Don'ts
+
+- Never use `args.join(' ')` before interpolation
+- Never use `raw()` with user input
+- Don't forget `await` on commands
+- Don't assume success without checking
+- Don't ignore stderr output
+
+---
+
+## See Also
+
+- [README.md](../README.md) - Main documentation
+- [docs/case-studies/issue-153/README.md](./docs/case-studies/issue-153/README.md) - Array.join() pitfall case study
+- [src/$.quote.mjs](./src/$.quote.mjs) - Quote function implementation

package/js/docs/IMPLEMENTATION_NOTES.md

@@ -0,0 +1,129 @@
+# Implementation Notes for Shell Operators Support
+
+## Summary
+
+The current implementation passes commands with `&&`, `||`, `;`, `()` to `sh -c`, which runs them in a subprocess. This prevents the virtual `cd` command from affecting the parent process directory.
+
+## Solution
+
+We've created a shell parser (`src/shell-parser.mjs`) that can parse these operators. Now we need to integrate it into `src/$.mjs` to execute parsed commands through our virtual command system.
+
+## Required Changes in src/$.mjs
+
+### 1. Import the parser
+
+```javascript
+import { parseShellCommand, needsRealShell } from './shell-parser.mjs';
+```
+
+### 2. Modify \_doStartAsync method
+
+Before checking for pipes, check if the command contains operators we can handle:
+
+```javascript
+async _doStartAsync() {
+  // ... existing code ...
+
+  // Check if command needs parsing for operators
+  if (this.spec.mode === 'shell' && !needsRealShell(this.spec.command)) {
+    const parsed = parseShellCommand(this.spec.command);
+    if (parsed && parsed.type === 'sequence') {
+      return await this._runSequence(parsed);
+    } else if (parsed && parsed.type === 'subshell') {
+      return await this._runSubshell(parsed);
+    }
+  }
+
+  // ... continue with existing pipeline check ...
+}
+```
+
+### 3. Add \_runSequence method
+
+```javascript
+async _runSequence(sequence) {
+  let lastResult = { code: 0, stdout: '', stderr: '' };
+  let combinedStdout = '';
+  let combinedStderr = '';
+
+  for (let i = 0; i < sequence.commands.length; i++) {
+    const command = sequence.commands[i];
+    const operator = i > 0 ? sequence.operators[i - 1] : null;
+
+    // Check operator conditions
+    if (operator === '&&' && lastResult.code !== 0) continue;
+    if (operator === '||' && lastResult.code === 0) continue;
+
+    // Execute command
+    if (command.type === 'subshell') {
+      lastResult = await this._runSubshell(command);
+    } else if (command.type === 'pipeline') {
+      lastResult = await this._runPipeline(command.commands);
+    } else if (command.type === 'simple') {
+      lastResult = await this._runSimpleCommand(command);
+    }
+
+    combinedStdout += lastResult.stdout;
+    combinedStderr += lastResult.stderr;
+  }
+
+  return {
+    code: lastResult.code,
+    stdout: combinedStdout,
+    stderr: combinedStderr
+  };
+}
+```
+
+### 4. Add \_runSubshell method
+
+```javascript
+async _runSubshell(subshell) {
+  // Save current directory
+  const savedCwd = process.cwd();
+
+  try {
+    // Execute subshell command
+    const result = await this._runSequence(subshell.command);
+    return result;
+  } finally {
+    // Restore directory
+    process.chdir(savedCwd);
+  }
+}
+```
+
+### 5. Add \_runSimpleCommand method
+
+```javascript
+async _runSimpleCommand(command) {
+  const { cmd, args, redirects } = command;
+
+  // Check for virtual command
+  if (virtualCommandsEnabled && virtualCommands.has(cmd)) {
+    const argValues = args.map(a => a.value);
+    return await this._runVirtual(cmd, argValues);
+  }
+
+  // Fall back to real command execution
+  // ... handle redirects if present ...
+
+  return await this._executeRealCommand(cmd, args);
+}
+```
+
+## Testing
+
+After implementation:
+
+1. `cd /tmp && pwd` should output `/tmp`
+2. `cd /tmp` followed by `pwd` should output `/tmp`
+3. `(cd /tmp && pwd) ; pwd` should output `/tmp` then original directory
+4. All existing tests should still pass
+
+## Benefits
+
+1. Virtual commands work with shell operators
+2. `cd` behaves exactly like shell cd
+3. Better performance (no subprocess for simple operator chains)
+4. Consistent behavior across platforms

package/js/docs/SHELL_OPERATORS_IMPLEMENTATION.md

@@ -0,0 +1,101 @@
+# Shell Operators Implementation Complete
+
+## Summary
+
+We've successfully implemented support for shell operators (`&&`, `||`, `;`, `()`) in command-stream, allowing virtual commands like `cd` to work correctly with these operators.
+
+## What Was Implemented
+
+### 1. Shell Parser (`src/shell-parser.mjs`)
+
+- Tokenizes and parses shell commands with operators
+- Handles `&&` (AND), `||` (OR), `;` (semicolon), `()` (subshells)
+- Supports pipes `|`, redirections `>`, `>>`, `<`
+- Properly handles quoted strings and escaped characters
+- Falls back to `sh -c` for unsupported features (globs, variable expansion, etc.)
+
+### 2. ProcessRunner Enhancements (`src/$.mjs`)
+
+- `_runSequence()` - Executes command sequences with proper operator semantics
+- `_runSubshell()` - Handles subshell isolation (saves/restores cwd)
+- `_runSimpleCommand()` - Executes individual commands (virtual or real)
+- Integration with existing pipeline support
+
+### 3. Fixed cd Command Behavior
+
+- `cd` now works correctly in all contexts:
+  - `cd /tmp` - changes directory (persists) ✓
+  - `cd /tmp && ls` - both commands see /tmp ✓
+  - `(cd /tmp && ls)` - subshell isolation ✓
+  - `cd /tmp ; pwd ; cd /usr ; pwd` - sequential execution ✓
+
+## How It Works
+
+1. When a command contains operators, the enhanced parser parses it into an AST
+2. The executor traverses the AST, respecting operator semantics:
+   - `&&` - run next only if previous succeeds (exit code 0)
+   - `||` - run next only if previous fails (exit code ≠ 0)
+   - `;` - run next regardless
+   - `()` - run in subshell with saved/restored directory
+3. Virtual commands execute in-process, maintaining state
+4. Real commands spawn subprocesses as needed
+5. Falls back to `sh -c` for unsupported features
+
+## Examples That Now Work
+
+```javascript
+// cd with && operator
+await $`cd /tmp && pwd`; // Output: /tmp
+
+// cd with || operator
+await $`cd /nonexistent || echo "failed"`; // Output: failed
+
+// Multiple commands with ;
+await $`cd /tmp ; pwd ; cd /usr ; pwd`; // Output: /tmp\n/usr
+
+// Subshell isolation
+await $`(cd /tmp && pwd) ; pwd`; // Output: /tmp\n<original-dir>
+
+// Complex chains
+await $`cd /tmp && git init && echo "done"`;
+
+// Nested subshells
+await $`(cd /tmp && (cd /usr && pwd) && pwd)`; // Output: /usr\n/tmp
+```
+
+## Benefits
+
+1. **Correct Shell Semantics** - cd and other virtual commands behave exactly like in a real shell
+2. **Performance** - No subprocess overhead for simple command chains
+3. **Cross-platform** - Consistent behavior across platforms
+4. **Backward Compatible** - Existing code continues to work
+5. **Graceful Fallback** - Complex shell features still work via `sh -c`
+
+## Testing
+
+All major shell operator scenarios are tested and working:
+
+- ✓ AND operator (`&&`)
+- ✓ OR operator (`||`)
+- ✓ Semicolon operator (`;`)
+- ✓ Subshells (`()`)
+- ✓ Nested subshells
+- ✓ Complex command chains
+- ✓ Directory persistence
+- ✓ Path with spaces
+
+## Files Modified
+
+1. `src/$.mjs` - Added sequence/subshell execution methods
+2. `src/shell-parser.mjs` - New file for parsing shell operators
+3. `src/commands/$.pwd.mjs` - Fixed to output newline
+4. Various test and example files
+
+## Future Enhancements
+
+Possible future additions:
+
+- Background execution (`&`)
+- Here documents (`<<`)
+- Process substitution (`<()`, `>()`)
+- More complex redirections (`2>&1`, etc.)