@litfamily/litcodex 1.0.0

package/marketplace/plugins/litcodex/skills/debugging/references/runtimes/native-binary.md

@@ -0,0 +1,484 @@
+# Native Binary Debugging (No Source / Reverse Engineering)
+
+For binaries where you don't have trustworthy source: stripped production builds, third-party closed libs, malware, CTF challenges, firmware, vendored libs whose docs lie. The workflow is specific; doing it out of order wastes days.
+
+This reference **coordinates** the triage and dynamic work. The heavy tools each have their own reference:
+- **Static decompilation** → [tools/ghidra.md](../tools/ghidra.md)
+- **Interactive debugging** → [tools/pwndbg.md](../tools/pwndbg.md)
+- **Scripted interaction / exploitation** → [tools/pwntools.md](../tools/pwntools.md)
+
+Read those before using them — especially Ghidra, which has a surprising amount of workflow that's not obvious.
+
+---
+
+## ⚠️ STOP — is this actually a stripped C/C++ binary?
+
+A growing share of "binaries" are actually **bundled high-level apps** — Bun SEA, Node SEA, Deno compile, pkg, nexe, Electron, Tauri, PyInstaller. Their workflow is completely different: the high-level source is recoverable with the right per-bundler tool (often plaintext, sometimes V8 cache / `.pyc` / eszip needing extra tooling), and Ghidra against the runtime VM wastes hours.
+
+Quick check:
+
+```bash
+file ./target                                                    # Mach-O / ELF / PE - inconclusive
+du -h ./target                                                   # 50 MB+ for a "simple CLI" → suspect bundled
+strings -n 12 ./target | rg -iE 'bun|node_modules|webpack|esbuild|deno|pkg/lib|electron|pyinstaller|nexe|NODE_SEA_FUSE|tauri' | head -5
+```
+
+**If any hits** → close this file, open [bundled-js-binary.md](bundled-js-binary.md) instead. Following the Ghidra/pwndbg path on a bundled-app binary wastes hours decompiling the runtime VM while the app-level bundle is recoverable with the right per-bundler tool (plaintext for Bun/pkg/nexe/Electron-asar; eszip / V8-cache / `.pyc` for Deno / Node SEA / PyInstaller).
+
+If `file` says "Mach-O" or "ELF", `du` is < 20 MB, and the strings check is empty → continue here.
+
+---
+
+## The workflow (do these in order)
+
+Every step's output is input to the next. Skipping steps means guessing later.
+
+```
+  [1] Triage           →  what kind of binary is this?
+  [2] Dynamic tracing  →  what syscalls / libcalls does it make?
+  [3] Static analysis  →  what does it DO, in readable form? (Ghidra)
+  [4] Dynamic debug    →  confirm hypotheses at runtime (pwndbg)
+  [5] Scripted repro   →  lock the bug with a pwntools script
+  [6] TDD + fix / report
+```
+
+Steps 1 and 2 are fast (minutes). Step 3 is slow (tens of minutes to hours depending on size). Don't skip 1-2 and go straight to Ghidra — the triage output tells you what to focus on inside Ghidra.
+
+---
+
+## [1] Triage — 5-minute fingerprint
+
+```bash
+# Basic identity
+file ./target
+#   elf, mach-o, pe? 32/64-bit? dynamically linked? stripped?
+
+# Architecture details
+readelf -h ./target                    # ELF header: entry point, arch, type
+lipo -info ./target 2>/dev/null        # macOS: universal binary?
+
+# Interesting strings (often leaks function names, error messages, URLs, API keys)
+strings -n 8 ./target | head -100
+strings -n 8 ./target | grep -iE '(http|/api/|error|debug|version)'
+
+# Imported symbols (what does it link against?)
+nm -D ./target 2>/dev/null              # dynamic symbols
+objdump -T ./target 2>/dev/null         # same, alternate tool
+readelf -d ./target                     # dynamic section (NEEDED libs)
+ldd ./target 2>/dev/null                # resolved library paths
+
+# Security posture (affects what exploits / bugs are possible)
+checksec --file=./target                # requires pwntools or installing checksec
+#   NX, PIE, RELRO, stack canary, FORTIFY
+
+# Is it stripped?
+nm ./target 2>/dev/null | head          # empty? stripped. full? not stripped.
+file ./target                           # will say "stripped" or "not stripped"
+```
+
+### ⚠️ `strings -n N` silently drops short content
+
+`strings` prints runs of printable characters of length **≥ N**. With `-n 8`, **anything shorter than 8 chars sandwiched between non-printable bytes is dropped silently**. This includes:
+
+- Short identifier interpolations in templates (`${x}`, `${i}`, `${R}`)
+- Short embedded constants (`v3`, `null`, integer immediates as bytes)
+- Short error codes between binary padding
+
+Real example: a JavaScript template literal `<INSTRUCTIONS>\n${x}\n</INSTRUCTIONS>` came out of `strings -n 8` as `<INSTRUCTIONS>\n</INSTRUCTIONS>` — the `${x}` (4 chars) was dropped. A consumer reading the dump would conclude the template was empty. It is not.
+
+**Use `strings` only for fingerprinting (Phase 1).** For any extraction whose correctness matters, **read bytes directly**:
+
+```bash
+# Count occurrences of a needle
+LC_ALL=C grep -aoc 'NEEDLE' ./target
+
+# Find offsets
+LC_ALL=C grep -aob 'NEEDLE' ./target | head
+
+# Or via Python for byte-precise context
+python3 -c "
+import sys
+data = open('./target','rb').read()
+needle = b'NEEDLE'
+pos = data.find(needle)
+print(repr(data[max(0,pos-100):pos+200]))
+"
+```
+
+If you must keep using `strings`, lower the threshold: `strings -n 1 -t x ./target | rg ...`. The signal-to-noise drops sharply but short content is preserved.
+
+Write the triage summary to the journal:
+
+```markdown
+## Binary triage
+- Type: <ELF 64-bit, dynamically linked, stripped>
+- Arch: <x86_64 | arm64 | ...>
+- Libs: <libc, openssl, libcurl>
+- Security: <NX, PIE, Partial RELRO, no canary>
+- Interesting strings: <short list>
+- First hypothesis surface: <which function / area looks most relevant>
+```
+
+---
+
+## [2] Dynamic tracing — what does it actually call?
+
+These are cheap — run them before Ghidra to orient yourself.
+
+### Linux: strace + ltrace
+
+```bash
+# System calls
+strace -f -o trace.out ./target arg1 arg2
+strace -f -e trace=network ./target           # filter to network syscalls
+strace -f -e trace=file ./target              # filter to file ops
+
+# Library calls (less useful when stripped but still informative)
+ltrace -f -o ltrace.out ./target
+ltrace -f -e 'str*+mem*' ./target             # filter to string/mem functions
+```
+
+### macOS: Mach-O specifics
+
+**SIP block reality check.** With System Integrity Protection enabled (default on every modern macOS), `dtruss` / `dtrace` will **silently fail** to attach to:
+- Anything in `/usr`, `/bin`, `/sbin`, `/System`
+- Apple-signed binaries (Xcode CLT, Homebrew formulae from Apple-distributed taps)
+- Notarized vendor binaries (Bun, Deno, Docker Desktop, etc.)
+
+`dtruss ./target` will appear to run but produce zero events. This is not a bug; it is the SIP design. Disabling SIP requires a Recovery Mode reboot — usually not worth it. Use the alternatives below.
+
+```bash
+# dtruss — works only when SIP allows it (your own unsigned binaries)
+sudo dtruss -f ./target 2>&1 | head -20         # equivalent to strace
+# If output is suspiciously empty → SIP blocked it. Switch to lldb or app-level logging.
+```
+
+**Mach-O metadata inspection (no SIP issues, no debugger needed):**
+
+```bash
+# Architecture and slices
+file ./target                                    # arm64 / x86_64 / universal
+lipo -info ./target                              # which architectures included
+lipo -thin arm64 ./target -output ./target-arm64 # extract one slice for analysis
+
+# Headers & load commands (segments, dylibs, code-signature pointer)
+otool -h ./target                                # Mach header (cputype, ncmds, flags)
+otool -l ./target | head -100                    # load commands; entitlements live in code-signature blob, see codesign below
+
+# Dynamic library dependencies (macOS equivalent of ldd)
+otool -L ./target                                # linked dylibs with versions
+dyld_info ./target                               # macOS 13+, more detailed than otool -L
+
+# Disassembly
+otool -tv ./target | head -200                   # quick disassembly without Ghidra
+otool -tV ./target                               # with symbol-resolved branches
+
+# Imported / exported symbols (Apple `nm`, NOT GNU)
+nm -u ./target                                   # undefined references = imports
+nm -gU ./target                                  # external defined = exports
+# Note: GNU `-D`/dynamic flags are not honored on Apple `nm`; use the above forms.
+symbols -fullSourcePath -onlyWithDebugInfo ./target  # if any debug info survives
+
+# Code signature & entitlements (entitlements come from codesign, NOT otool)
+codesign -dv --entitlements :- ./target 2>&1     # signature info + entitlements XML on stdout
+spctl --assess --type execute -vv ./target       # Gatekeeper assessment
+
+# Cert chain — extract to a temp dir to avoid creating files named -0/-1 in cwd
+tmp=$(mktemp -d)
+codesign -dvv --extract-certificates="$tmp/cert" ./target 2>&1
+ls -la "$tmp"
+# rm -rf "$tmp"  # journal first, clean up later
+
+# Strings inside specific segments only (less noise than full-binary strings)
+otool -s __TEXT __cstring ./target               # C string section
+otool -s __TEXT __const ./target                 # constants section
+```
+
+**Interactive debugging on macOS — use `lldb`, not `gdb`.**
+
+GDB on macOS requires a self-signed code-signing certificate (`codesign --entitlements gdb.entitlements --sign gdb-cert /opt/homebrew/bin/gdb`) and even then is unreliable on arm64. **Use `lldb` directly** — it ships with Xcode CLT and works without configuration.
+
+```bash
+# Start lldb
+lldb ./target
+
+# Set arguments
+(lldb) settings set target.run-args arg1 arg2
+
+# Run with breakpoints
+(lldb) breakpoint set --name function_name        # symbol-based
+(lldb) breakpoint set --address 0x1000034c0       # address-based
+(lldb) breakpoint set --regex '.*decode.*'         # regex over symbols
+
+# Run / step / inspect
+(lldb) run
+(lldb) bt                                          # backtrace
+(lldb) frame variable                              # locals
+(lldb) register read                               # all registers
+(lldb) memory read --size 8 --format x --count 16 $sp   # 16 qwords from stack
+(lldb) disassemble --frame                         # current function
+(lldb) image list                                  # loaded modules
+(lldb) image lookup -a 0x1000034c0                 # which module + symbol owns this address
+
+# Process attach to running process
+(lldb) process attach --pid 12345
+(lldb) process attach --name target               # attach by name
+
+# Print Mach-O specific
+(lldb) image dump sections ./target
+(lldb) image dump symtab ./target
+```
+
+**Function interception via `DYLD_INSERT_LIBRARIES`** (macOS equivalent of `LD_PRELOAD`):
+
+```bash
+# Build a shim dylib that overrides specific functions
+# Then run target with it preloaded
+DYLD_INSERT_LIBRARIES=./shim.dylib DYLD_FORCE_FLAT_NAMESPACE=1 ./target
+```
+
+DYLD_INSERT works in the unrestricted case but is blocked in three distinct scenarios — distinguish them when diagnosing why your shim didn't load:
+
+1. **SIP / restricted process** (target has the `__RESTRICT,__restrict` section, is setuid/setgid, or is a platform/Apple-signed binary): dyld unconditionally strips all `DYLD_*` env vars before the process starts. Nothing you set will reach the target.
+2. **Hardened runtime + library validation** (`CS_RUNTIME` flag set, `com.apple.security.cs.disable-library-validation` entitlement absent): the process accepts `DYLD_INSERT_LIBRARIES` but **rejects** loading any dylib that isn't signed by the same Team ID or by Apple. Symptom: shim is found but not loaded; check `log show --predicate 'eventMessage CONTAINS "library validation failed"'`.
+3. **Notarization / Gatekeeper translocation**: the binary may be running from a translocated path; relative paths in `DYLD_INSERT_LIBRARIES` won't resolve. Use absolute paths.
+
+Check each:
+
+```bash
+# Restrict segment present? (case 1)
+otool -l ./target | grep -A2 __RESTRICT
+# Hardened runtime flag? (case 2)
+codesign -d --verbose=4 ./target 2>&1 | grep -iE 'flags=|CodeDirectory'
+# Look for "0x10000(runtime)" or similar in the flags line.
+# Disable-library-validation entitlement?
+codesign -d --entitlements :- ./target 2>&1 | grep disable-library-validation
+```
+
+**App-level debug logging (always works, ignores SIP):**
+
+When debugger attach is blocked, fall back to maximizing the app's own logging:
+
+```bash
+# Try common patterns
+APP_DEBUG=1 APP_LOG_LEVEL=debug APP_LOG_FILE=/tmp/trace.log ./target
+NSDebugEnabled=YES ./target                       # Cocoa apps
+OS_ACTIVITY_MODE=debug ./target                   # os_log subsystem
+
+# Then read os_log unified logging stream live
+log stream --predicate 'process == "target"' --level debug
+
+# Or extract historical logs
+log show --predicate 'process == "target"' --last 1h --info --debug
+```
+
+This is the **partial-runtime-evidence path** for macOS. See [methodology/partial-runtime-evidence.md](../methodology/partial-runtime-evidence.md) for how to combine app-level logs with static analysis when wire-level capture is blocked.
+
+**Network capture on macOS (TLS-decrypted):**
+
+```bash
+# 1. Find the active network service (don't assume "Wi-Fi"):
+#    Map the default-route interface to the matching networksetup service name.
+networksetup -listallnetworkservices                        # show options
+DEFAULT_IF=$(route -n get default 2>/dev/null | awk '/interface:/ {print $2}')
+echo "Default-route interface: $DEFAULT_IF"
+# Match the interface (en0, en1, ...) back to a service name:
+SERVICE=$(networksetup -listallhardwareports | awk -v iface="$DEFAULT_IF" '
+  /^Hardware Port:/ { hp = substr($0, index($0,$3)) }
+  /^Device:/        { if ($2 == iface) print hp }
+')
+if [ -z "$SERVICE" ]; then
+  echo "Could not auto-detect active service. Pick one from -listallnetworkservices manually." >&2
+  echo "Aborting proxy setup." >&2
+  false  # signal failure but stay safe at top level
+else
+  echo "Using service: $SERVICE"
+fi
+
+# 2. JOURNAL the original proxy state before changing it (REQUIRED for safe rollback):
+networksetup -getwebproxy "$SERVICE"        # save this output to journal
+networksetup -getsecurewebproxy "$SERVICE"  # save this too
+
+# 3. Start mitmproxy with persistent CA at ~/.mitmproxy/
+mitmproxy --listen-host 127.0.0.1 --listen-port 8888 &
+
+# 4. Trust the mitmproxy CA system-wide if the target uses URLSession or any framework
+# that ignores HTTPS_PROXY/SSL_CERT_FILE (most macOS-native apps do):
+sudo security add-trusted-cert -d -r trustRoot -k /Library/Keychains/System.keychain ~/.mitmproxy/mitmproxy-ca-cert.pem
+
+# 5. Two routing options. Try env-var first; fall back to system proxy:
+# 5a. Apps that honor env vars (most CLIs):
+HTTPS_PROXY=http://127.0.0.1:8888 SSL_CERT_FILE=~/.mitmproxy/mitmproxy-ca-cert.pem ./target ...
+
+# 5b. Apps that use URLSession / system network config (most GUI apps, Bun, some CLIs):
+networksetup -setwebproxy "$SERVICE" 127.0.0.1 8888
+networksetup -setsecurewebproxy "$SERVICE" 127.0.0.1 8888
+
+# 6. Cleanup — RESTORE original state from journal, untrust CA:
+networksetup -setwebproxystate "$SERVICE" off
+networksetup -setsecurewebproxystate "$SERVICE" off
+sudo security delete-certificate -c "mitmproxy" /Library/Keychains/System.keychain
+```
+
+**Critical**: forgetting step 6 leaves all your subsequent traffic mis-routed and silently MITM-able. Journal every step.
+
+### What to look for
+
+| Observation | Hypothesis |
+|---|---|
+| `open("/etc/secret-config", ...)` | Reads unexpected config; look at what it does with contents |
+| `connect(... 1.2.3.4:443)` | Phones home or depends on an external service |
+| `getenv("FOO")` returning NULL | Env var expected but not set |
+| Repeated `poll`/`epoll_wait` with no progress | Stuck on I/O; check downstream |
+| `SIGSEGV` caught by signal handler | Custom crash recovery — often hides the real bug |
+| `dlopen("libfoo.so.42")` | Dynamic plugin loading; check plugin path |
+
+---
+
+## [3] Static analysis with Ghidra
+
+When triage + tracing have narrowed you to "something in function X" or "the crypto routine is weird", open Ghidra.
+
+**Open [tools/ghidra.md](../tools/ghidra.md) before launching Ghidra** — the import / analyze / decompile workflow is not obvious and first-time users waste an hour figuring it out.
+
+Ghidra's decompiler turns machine code into readable-ish C. That's usually what you want. Stay in the Decompiler view; drop to Listing (disassembly) only when the decompiler punts.
+
+---
+
+## [4] Dynamic debugging with pwndbg
+
+Once static analysis gives you a hypothesis ("this branch at 0x401234 is where the validation fails"), confirm it at runtime with pwndbg.
+
+**Open [tools/pwndbg.md](../tools/pwndbg.md) before launching gdb.** Pwndbg gives you the context view (registers / stack / disasm / code all visible at once) which is essential for binary debugging.
+
+Typical pwndbg flow:
+
+```
+$ gdb ./target                                 # pwndbg loads automatically if installed
+pwndbg> break *0x401234                        # break at the address static analysis flagged
+pwndbg> run arg1 arg2
+# At the breakpoint:
+pwndbg> context                                # registers + stack + disasm
+pwndbg> telescope $rdi                         # walk pointers at $rdi
+pwndbg> x/20xw $rsp                            # raw dump of stack
+pwndbg> ni / si                                # step next / step instruction
+```
+
+---
+
+## [5] Scripted reproduction with pwntools
+
+Once you have a hypothesis with a concrete repro input, lock it down with pwntools. This is the "failing test" equivalent for binaries.
+
+**Open [tools/pwntools.md](../tools/pwntools.md)** — the Process/Remote/ELF/context APIs are the foundation.
+
+```python
+from pwn import *
+
+context.binary = elf = ELF('./target')
+
+p = process('./target')
+p.sendlineafter(b'> ', b'<trigger input that reproduces the bug>')
+result = p.recvall(timeout=3)
+assert b'expected-output-when-fixed' in result, f'bug repro: {result}'
+```
+
+This script is now your "red test". When the fix is applied, the script should pass (or the assertion should be inverted for negative tests — e.g. "the crash string should NOT appear").
+
+---
+
+## [6] Fixing a binary bug you can't recompile
+
+Three options, in preference order:
+
+### Option A: Patch at the source (if you have it)
+
+If the bug is in your own code and source is available, fix it there and rebuild. Standard TDD path.
+
+### Option B: Binary patch
+
+For tiny fixes (one byte, one branch inversion):
+
+```bash
+# Identify the exact byte offset
+# e.g. Ghidra says the bug is at 0x401234 = file offset 0x1234
+printf '\x90\x90' | dd of=./target bs=1 seek=$((0x1234)) conv=notrunc
+```
+
+Journal the exact `dd` command and the original bytes so you can revert.
+
+### Option C: Wrap / shim
+
+If you can't patch the binary, write a shim library (LD_PRELOAD on Linux, DYLD_INSERT_LIBRARIES on macOS) that overrides the buggy function. pwntools has examples.
+
+### Option D: Report upstream
+
+If it's a third-party binary and none of the above are feasible, the "fix" is a high-quality bug report with:
+- Full triage summary
+- Reproducible pwntools script
+- Ghidra decompilation of the buggy function
+- Hypothesis about the root cause
+- Recommended patch sketch (in C or pseudocode)
+
+---
+
+## Silent-failure patterns in native binaries
+
+| Pattern | Why it's silent |
+|---|---|
+| Ignored libc return codes (`read`, `write`, `malloc`) | Bug continues with garbage data; no check |
+| Signal handler swallows SIGSEGV | Crash converted to "something didn't work"; no log |
+| `setjmp`/`longjmp` unwinding over cleanup | Resources leak silently |
+| Thread-local error state never read (`errno`, `GetLastError`) | Error happened, nobody asked |
+| Recovered assertion failure in release build | `assert` compiled out; precondition violations silently corrupt |
+| Dangling pointer reads after free | Often looks like valid data until it doesn't |
+
+---
+
+## Phase 9 cleanup specifics
+
+```bash
+# Kill debugger sessions
+pkill -f 'gdb' || true
+pkill -f 'lldb' || true
+
+# Ghidra scratch projects (if made just for this session)
+# Named something like ~/ghidra-projects/debug-<timestamp>:
+ls -la ~/ghidra-projects/ 2>/dev/null
+# rm -rf ~/ghidra-projects/debug-scratch         # only if the journal says to
+
+# Core dumps left from crashes
+rm -f ./core ./core.* ~/core.*
+
+# strace/ltrace output files
+rm -f trace.out ltrace.out
+
+# If you made a binary patch (Option B above), confirm revert
+# The journal should have the original bytes — restore them:
+# printf '<original-bytes>' | dd of=./target bs=1 seek=<offset> conv=notrunc
+
+# Trace-output files
+rm -f /tmp/debug-*.bin /tmp/debug-*.strace /tmp/debug-*.ltrace
+
+# macOS-specific:
+# Restore proxy settings if you set them (CRITICAL — leaves system traffic mis-routed otherwise)
+# Use the SAME $SERVICE you used when enabling the proxy (read it from the journal).
+# Do NOT hardcode "Wi-Fi" — many machines route traffic over Ethernet, USB tether, or a VPN service.
+[ -n "$SERVICE" ] && {
+  networksetup -setwebproxystate "$SERVICE" off 2>/dev/null
+  networksetup -setsecurewebproxystate "$SERVICE" off 2>/dev/null
+}
+# Or restore explicitly from the journaled original state — see the proxy section above.
+
+# Stop mitmproxy
+pkill -f 'mitmproxy' 2>/dev/null
+
+# Remove DYLD shim libraries you built
+rm -f /tmp/*-shim.dylib
+
+# Clear extracted strings dumps (these can be huge and may contain secrets)
+rm -f /tmp/*-strings*.txt
+
+# Verify hostname resolution returns to normal (mitmproxy can leave entries)
+scutil --dns | head -20
+```