@hybridlabor-api/aos 4.8.0 → 4.9.0

package/skills/global_config/makefile-generator/docs/optimization-guide.md

@@ -0,0 +1,784 @@
+# Makefile Optimization Guide
+
+## Overview
+
+This guide covers techniques for optimizing Makefile performance, including parallel builds, dependency tracking, incremental builds, caching strategies, and performance profiling.
+
+## Parallel Builds
+
+### Enabling Parallel Execution
+
+```bash
+# Run with 4 parallel jobs
+make -j4
+
+# Use all CPU cores
+make -j$(nproc)
+
+# Unlimited parallel jobs (careful!)
+make -j
+```
+
+### Making Makefiles Parallel-Safe
+
+**Problem: Shared resources**
+
+```makefile
+# WRONG: Multiple rules write to same file
+target1:
+	echo "data1" >> shared.log
+
+target2:
+	echo "data2" >> shared.log
+
+# With -j2, file corruption likely!
+```
+
+**Solution: Proper dependencies**
+
+```makefile
+# RIGHT: Serialize access with dependencies
+target2: target1
+
+# Or use separate files
+target1:
+	echo "data1" > target1.log
+
+target2:
+	echo "data2" > target2.log
+```
+
+### Controlling Parallelism
+
+```makefile
+# Disable parallel builds for this Makefile
+.NOTPARALLEL:
+
+# Disable parallelism for specific targets
+.NOTPARALLEL: install clean
+
+# Serialize specific targets
+install: build
+	# Install runs after build completes
+```
+
+### GNU Make 4.4+ Parallel Control Features
+
+GNU Make 4.4 (released October 2022) introduced new features for fine-grained parallel control. These are becoming part of the upcoming POSIX standard.
+
+#### .WAIT Special Target
+
+The `.WAIT` target provides explicit ordering without creating artificial dependencies:
+
+```makefile
+# .WAIT ensures prerequisites to its left complete
+# before starting prerequisites to its right
+all: compile .WAIT link .WAIT package
+
+# Equivalent behavior without .WAIT would require:
+# link: compile
+# package: link
+# But .WAIT is cleaner when targets are independent conceptually
+```
+
+**Use Cases for .WAIT:**
+
+```makefile
+# Build phases with explicit ordering
+build: setup .WAIT compile .WAIT test .WAIT package
+	@echo "Build complete"
+
+# Parallel within phases, serial between phases
+ci: lint fmt .WAIT test.unit test.integration .WAIT build
+# lint and fmt run in parallel
+# then unit and integration tests run in parallel
+# finally build runs
+
+# Database migrations before tests
+test: migrate .WAIT run-tests
+	@echo "Tests complete"
+```
+
+**Important Notes:**
+- `.WAIT` only affects parallel builds (`make -j`)
+- With sequential execution, `.WAIT` has no effect
+- `.WAIT` doesn't create actual dependencies, just ordering
+- Available in GNU Make 4.4+ (check with `make --version`)
+
+#### .NOTPARALLEL with Prerequisites (Enhanced)
+
+In Make 4.4+, `.NOTPARALLEL` can take specific targets as prerequisites:
+
+```makefile
+# Traditional: Disable ALL parallel execution
+.NOTPARALLEL:
+
+# NEW in 4.4: Serialize only specific targets
+.NOTPARALLEL: install deploy cleanup
+
+# This implicitly adds .WAIT between each prerequisite
+# of the listed targets
+```
+
+**When to Use .NOTPARALLEL with Prerequisites:**
+
+```makefile
+# Deployment must be serial (avoid race conditions)
+.NOTPARALLEL: deploy
+
+deploy: deploy-database deploy-backend deploy-frontend
+	@echo "Deployment complete"
+# deploy-database -> deploy-backend -> deploy-frontend (serial)
+
+# But compilation can still be parallel
+build: $(OBJECTS)
+	$(CC) $^ -o $(TARGET)
+# Object files compile in parallel (unaffected by .NOTPARALLEL: deploy)
+```
+
+#### Version Checking for Make 4.4+ Features
+
+Check Make version before using 4.4+ features:
+
+```makefile
+# Check Make version (4.4 = 4.4, need >= 4.4)
+MAKE_VERSION_MAJOR := $(word 1,$(subst ., ,$(MAKE_VERSION)))
+MAKE_VERSION_MINOR := $(word 2,$(subst ., ,$(MAKE_VERSION)))
+
+# Simple version check
+ifeq ($(shell expr $(MAKE_VERSION_MAJOR) \>= 4),1)
+ifeq ($(shell expr $(MAKE_VERSION_MINOR) \>= 4),1)
+    HAVE_WAIT := 1
+endif
+endif
+
+# Alternative: Graceful degradation
+ifdef HAVE_WAIT
+    # Use .WAIT for modern Make
+    all: compile .WAIT link
+else
+    # Fall back to dependencies for older Make
+    link: compile
+    all: link
+endif
+```
+
+**Practical Version Check Pattern:**
+
+```makefile
+# At the top of your Makefile
+MIN_MAKE_VERSION := 4.4
+CURRENT_MAKE_VERSION := $(MAKE_VERSION)
+
+# Check and warn if using older Make
+ifeq ($(shell printf '%s\n' "$(MIN_MAKE_VERSION)" "$(CURRENT_MAKE_VERSION)" | sort -V | head -n1),$(MIN_MAKE_VERSION))
+    # Make version is sufficient
+else
+    $(warning GNU Make $(MIN_MAKE_VERSION)+ recommended. You have $(CURRENT_MAKE_VERSION))
+    $(warning Some parallel control features may not work)
+endif
+```
+
+#### Comparison: .WAIT vs Dependencies vs .NOTPARALLEL
+
+| Feature | Use Case | Make Version |
+|---------|----------|--------------|
+| Dependencies (`b: a`) | Actual dependency relationship | All |
+| `.WAIT` | Ordering without dependency | 4.4+ |
+| `.NOTPARALLEL:` (global) | Disable all parallel | All |
+| `.NOTPARALLEL: target` | Serialize specific target's prereqs | 4.4+ |
+
+**Example Comparison:**
+
+```makefile
+# Using dependencies (works in all Make versions)
+# Problem: Creates false dependency relationship
+link: compile
+package: link
+all: package
+
+# Using .WAIT (Make 4.4+)
+# Cleaner: Explicit ordering, no false dependencies
+all: compile .WAIT link .WAIT package
+
+# Using .NOTPARALLEL with targets (Make 4.4+)
+# Best for: Targets that must never run in parallel
+.NOTPARALLEL: deploy
+deploy: step1 step2 step3
+```
+
+### Optimal Parallel Structure
+
+```makefile
+# Good parallel structure
+SOURCES := src1.c src2.c src3.c src4.c
+OBJECTS := $(SOURCES:.c=.o)
+
+# All .o files can build in parallel
+program: $(OBJECTS)
+	$(CC) $^ -o $@
+
+%.o: %.c
+	$(CC) -c $< -o $@
+```
+
+**Parallel execution:**
+```
+make -j4
+# Compiles 4 .c files simultaneously
+# Then links when all are done
+```
+
+## Dependency Tracking
+
+### Accurate Dependencies
+
+**Problem: Incorrect dependencies**
+
+```makefile
+# WRONG: Missing header dependencies
+main.o: main.c
+	$(CC) -c $< -o $@
+
+# If common.h changes, main.o won't rebuild!
+```
+
+**Solution: Automatic dependency generation**
+
+```makefile
+# Generate dependencies during compilation
+%.o: %.c
+	$(CC) $(CFLAGS) -MMD -MP -c $< -o $@
+
+# Include generated .d files
+-include $(OBJECTS:.o=.d)
+```
+
+**Generated dependency file (main.d):**
+```makefile
+main.o: main.c common.h utils.h
+common.h:
+utils.h:
+```
+
+### Dependency Flags
+
+```makefile
+# -MMD: Generate dependency file (.d)
+# -MP: Add phony targets for headers
+# -MF file: Specify dependency file name
+
+DEPFLAGS = -MMD -MP -MF $(@:.o=.d)
+
+%.o: %.c
+	$(CC) $(CFLAGS) $(DEPFLAGS) -c $< -o $@
+```
+
+### Why -MP is Important
+
+**Without -MP:**
+```makefile
+# Generated main.d:
+main.o: main.c utils.h
+
+# If utils.h is deleted:
+make: *** No rule to make target 'utils.h'. Stop.
+```
+
+**With -MP:**
+```makefile
+# Generated main.d:
+main.o: main.c utils.h
+utils.h:
+
+# If utils.h is deleted, make continues
+# (assumes you also removed #include "utils.h")
+```
+
+## Incremental Builds
+
+### Timestamp-Based Builds
+
+Make rebuilds targets when prerequisites are newer:
+
+```makefile
+# program rebuilt if any .o is newer
+program: $(OBJECTS)
+	$(CC) $^ -o $@
+
+# main.o rebuilt if main.c or headers are newer
+main.o: main.c common.h
+	$(CC) -c main.c -o main.o
+```
+
+### Optimizing Dependency Chains
+
+**Inefficient:**
+```makefile
+# Every source depends on config.h
+# Changing config.h rebuilds EVERYTHING
+main.o: main.c config.h
+utils.o: utils.c config.h
+helper.o: helper.c config.h
+```
+
+**Better: Only include where needed**
+```makefile
+# Only main.c actually uses config.h
+main.o: main.c config.h
+utils.o: utils.c
+helper.o: helper.c
+```
+
+**Best: Use automatic dependencies**
+```makefile
+%.o: %.c
+	$(CC) $(CFLAGS) -MMD -MP -c $< -o $@
+
+-include $(DEPENDS)
+# Automatically tracks which headers each file uses
+```
+
+### Intermediate File Management
+
+```makefile
+# Mark intermediate files
+.INTERMEDIATE: $(OBJECTS)
+# Deleted after use
+
+# Keep important intermediate files
+.SECONDARY: important.o
+# Not deleted
+
+# Never delete these files
+.PRECIOUS: %.o %.d
+# Protected from deletion
+```
+
+### Avoiding Unnecessary Rebuilds
+
+**Problem: Timestamp updates without changes**
+
+```makefile
+# WRONG: Always updates config.h
+config.h: config.h.in
+	sed 's/@VERSION@/$(VERSION)/g' $< > $@
+	# Updates timestamp even if content unchanged!
+```
+
+**Solution: Conditional update**
+
+```makefile
+# RIGHT: Only update if different
+config.h: config.h.in
+	sed 's/@VERSION@/$(VERSION)/g' $< > $@.tmp
+	cmp -s $@.tmp $@ || mv $@.tmp $@
+	rm -f $@.tmp
+```
+
+## Build Caching
+
+### Compiler Cache (ccache)
+
+```makefile
+# Use ccache for faster recompilation
+CC := ccache gcc
+CXX := ccache g++
+
+# Or conditionally:
+ifeq ($(shell command -v ccache 2>/dev/null),)
+CC ?= gcc
+else
+CC ?= ccache gcc
+endif
+```
+
+**Benefits:**
+- Caches compilation results
+- Speeds up clean rebuilds
+- Useful for CI/CD and switching branches
+
+### Distcc for Distributed Compilation
+
+```makefile
+# Distributed compilation across network
+CC := distcc gcc
+CXX := distcc g++
+
+# Set number of jobs based on available hosts
+DISTCC_HOSTS := localhost/2 build1/4 build2/4
+JOBS := 10
+```
+
+### Build Directory Caching
+
+```makefile
+# Keep build artifacts between clean builds
+.PHONY: clean distclean
+
+clean:
+	$(RM) $(TARGET)
+	# Keep .o and .d files for faster rebuild
+
+distclean: clean
+	$(RM) -r $(BUILDDIR)
+	# Complete clean
+```
+
+## Performance Optimization Techniques
+
+### 1. Use := Instead of =
+
+```makefile
+# SLOW: Recursive expansion (evaluated every use)
+SOURCES = $(wildcard src/*.c)
+OBJECTS = $(SOURCES:.c=.o)
+# $(OBJECTS) re-runs wildcard every time!
+
+# FAST: Simple expansion (evaluated once)
+SOURCES := $(wildcard src/*.c)
+OBJECTS := $(SOURCES:.c=.o)
+# Evaluated once when defined
+```
+
+### 2. Minimize Shell Invocations
+
+```makefile
+# SLOW: Multiple shell calls
+FILES = $(shell ls *.c)
+COUNT = $(shell ls *.c | wc -l)
+
+# FAST: Single shell call
+FILES := $(wildcard *.c)
+COUNT := $(words $(FILES))
+```
+
+### 3. Use Static Pattern Rules
+
+```makefile
+OBJECTS := main.o utils.o helper.o
+
+# FASTER: Static pattern rule (make knows exact files)
+$(OBJECTS): %.o: %.c
+	$(CC) -c $< -o $@
+
+# SLOWER: Pattern rule (make searches for matches)
+%.o: %.c
+	$(CC) -c $< -o $@
+```
+
+### 4. Reduce Makefile Parsing Time
+
+```makefile
+# SLOW: Complex shell commands in variable assignment
+VERSION = $(shell git describe --tags --always --dirty)
+
+# FAST: Use := to evaluate once
+VERSION := $(shell git describe --tags --always --dirty)
+
+# FASTER: Cache in file
+VERSION := $(file < VERSION.txt)
+```
+
+### 5. Avoid Recursive Make
+
+**Inefficient: Recursive Make**
+
+```makefile
+# Top-level Makefile
+SUBDIRS := lib1 lib2 app
+
+all:
+	for dir in $(SUBDIRS); do $(MAKE) -C $$dir; done
+```
+
+**Problems:**
+- Multiple make invocations (slow)
+- Incorrect dependency tracking
+- Parallel builds broken
+
+**Efficient: Non-Recursive Make**
+
+```makefile
+# Single Makefile
+LIB1_SRC := $(wildcard lib1/*.c)
+LIB2_SRC := $(wildcard lib2/*.c)
+APP_SRC := $(wildcard app/*.c)
+
+ALL_SRC := $(LIB1_SRC) $(LIB2_SRC) $(APP_SRC)
+OBJECTS := $(ALL_SRC:.c=.o)
+
+# Single dependency tree
+# Accurate parallel builds
+```
+
+**Reference:** "Recursive Make Considered Harmful" by Peter Miller
+
+## Performance Profiling
+
+### Timing Individual Targets
+
+```makefile
+# Time recipe execution
+%.o: %.c
+	@echo "Compiling $<..."
+	@time $(CC) $(CFLAGS) -c $< -o $@
+```
+
+### Build Time Measurement
+
+```bash
+# Time entire build
+time make -j4
+
+# Output:
+# real    0m12.345s
+# user    0m45.678s
+# sys     0m3.456s
+```
+
+### Debug Output for Performance Analysis
+
+```bash
+# Show what make is doing
+make -d
+
+# Show only remake decisions
+make -d --debug=basic
+
+# Show implicit rule search
+make -d --debug=implicit
+
+# Profile make itself
+make --profile=profile.log
+```
+
+### Finding Bottlenecks
+
+```makefile
+# Add timing to critical paths
+$(TARGET): $(OBJECTS)
+	@echo "==> Linking $(TARGET)"
+	@time $(CC) $(LDFLAGS) $^ $(LDLIBS) -o $@
+
+%.o: %.c
+	@echo "==> Compiling $<"
+	@time $(CC) $(CFLAGS) -c $< -o $@
+```
+
+## Optimization Best Practices
+
+### 1. Structure for Parallelism
+
+```makefile
+# Good: Independent compilation
+OBJECTS := a.o b.o c.o d.o
+
+program: $(OBJECTS)
+	$(CC) $^ -o $@
+
+%.o: %.c
+	$(CC) -c $< -o $@
+
+# make -j4 compiles 4 files at once
+```
+
+### 2. Accurate Dependencies
+
+```makefile
+# Use automatic dependency generation
+CFLAGS += -MMD -MP
+-include $(OBJECTS:.o=.d)
+
+# Not manual maintenance
+```
+
+### 3. Minimal Clean
+
+```makefile
+# Keep intermediate files by default
+clean:
+	$(RM) $(TARGET)
+
+# Full clean only when needed
+distclean: clean
+	$(RM) $(OBJECTS) $(DEPENDS)
+```
+
+### 4. Efficient Variable Usage
+
+```makefile
+# Use := for computed values
+SOURCES := $(wildcard src/*.c)
+OBJECTS := $(SOURCES:.c=.o)
+
+# Use ?= for user overrides
+CC ?= gcc
+CFLAGS ?= -O2
+```
+
+### 5. Avoid Unnecessary Work
+
+```makefile
+# Don't rebuild if nothing changed
+config.h: config.h.in Makefile
+	@sed 's/@VERSION@/$(VERSION)/g' $< > $@.tmp
+	@if ! cmp -s $@ $@.tmp; then \
+		echo "  GEN     $@"; \
+		mv $@.tmp $@; \
+	else \
+		rm -f $@.tmp; \
+	fi
+```
+
+## Complete Optimized Example
+
+```makefile
+# Optimized Makefile for C project
+.DELETE_ON_ERROR:
+.SUFFIXES:
+
+PROJECT := optimized
+VERSION := 1.0.0
+
+# User-overridable (use ?=)
+CC ?= gcc
+CFLAGS ?= -Wall -Wextra -O2
+PREFIX ?= /usr/local
+
+# Computed once (use :=)
+SRCDIR := src
+BUILDDIR := build
+OBJDIR := $(BUILDDIR)/obj
+
+SOURCES := $(wildcard $(SRCDIR)/*.c)
+OBJECTS := $(SOURCES:$(SRCDIR)/%.c=$(OBJDIR)/%.o)
+DEPENDS := $(OBJECTS:.o=.d)
+TARGET := $(BUILDDIR)/$(PROJECT)
+
+# Check for ccache
+ifneq ($(shell command -v ccache 2>/dev/null),)
+    CC := ccache $(CC)
+endif
+
+# Optimization flags for dependencies
+DEPFLAGS = -MMD -MP
+
+.PHONY: all clean distclean profile
+
+all: $(TARGET)
+
+# Link (serial)
+$(TARGET): $(OBJECTS)
+	@mkdir -p $(@D)
+	@echo "  LD      $@"
+	$(CC) $(LDFLAGS) $^ $(LDLIBS) -o $@
+
+# Compile (parallel-safe)
+$(OBJDIR)/%.o: $(SRCDIR)/%.c
+	@mkdir -p $(@D)
+	@echo "  CC      $<"
+	$(CC) $(CPPFLAGS) $(CFLAGS) $(DEPFLAGS) -c $< -o $@
+
+# Include auto-generated dependencies
+-include $(DEPENDS)
+
+# Minimal clean (keeps .o for faster rebuild)
+clean:
+	$(RM) $(TARGET)
+
+# Full clean
+distclean:
+	$(RM) -r $(BUILDDIR)
+
+# Profile build
+profile:
+	time $(MAKE) clean
+	time $(MAKE) -j$(shell nproc) all
+```
+
+## Benchmarking Results
+
+**Example project: 100 C files**
+
+| Configuration | Build Time | Rebuild Time |
+|---------------|------------|--------------|
+| Sequential (make) | 45s | 12s |
+| Parallel -j2 | 25s | 7s |
+| Parallel -j4 | 15s | 4s |
+| Parallel -j8 | 12s | 3s |
+| Parallel + ccache (cold) | 14s | 3s |
+| Parallel + ccache (warm) | 3s | 1s |
+
+**Key takeaways:**
+- Parallel builds: 3-4x speedup
+- ccache (warm): 10x speedup on clean builds
+- Accurate dependencies: Only rebuild what changed
+
+## Advanced Optimization
+
+### Precompiled Headers
+
+```makefile
+# Generate precompiled header
+$(OBJDIR)/common.h.gch: $(SRCDIR)/common.h
+	@mkdir -p $(@D)
+	$(CC) $(CPPFLAGS) $(CFLAGS) -x c-header $< -o $@
+
+# Use precompiled header
+$(OBJDIR)/%.o: $(SRCDIR)/%.c $(OBJDIR)/common.h.gch
+	$(CC) $(CPPFLAGS) $(CFLAGS) -include $(OBJDIR)/common.h -c $< -o $@
+```
+
+### Link-Time Optimization (LTO)
+
+```makefile
+# Enable LTO for release builds
+release: CFLAGS += -flto -O3
+release: LDFLAGS += -flto -O3
+release: $(TARGET)
+```
+
+### Unity Builds
+
+```makefile
+# Combine all sources into one compilation unit
+unity.c: $(SOURCES)
+	@echo "Generating unity build..."
+	@for src in $(SOURCES); do \
+		echo "#include \"$$src\"" >> $@; \
+	done
+
+unity.o: unity.c
+	$(CC) $(CFLAGS) -c $< -o $@
+
+# Fast single-file compilation
+# Trade-off: No parallel compilation
+```
+
+## Profiling Tools
+
+```bash
+# Make's built-in profiling
+make --profile=profile.log
+# Analyze profile.log
+
+# Time individual targets
+make -d 2>&1 | grep -E "Considering|Must remake"
+
+# strace for system call analysis
+strace -c make 2>&1 | tail -20
+
+# Remake (make debugger)
+remake --debug
+```
+
+## References
+
+- [GNU Make Manual - Parallel Execution](https://www.gnu.org/software/make/manual/html_node/Parallel.html)
+- [GNU Make Manual - Parallel Disable (.WAIT, .NOTPARALLEL)](https://www.gnu.org/software/make/manual/html_node/Parallel-Disable.html)
+- [GNU Make 4.4 Release Notes](https://lists.gnu.org/archive/html/info-gnu/2022-10/msg00008.html)
+- [GNU Make 4.4.1 Release Notes](https://lists.gnu.org/archive/html/info-gnu/2023-02/msg00011.html)
+- [Recursive Make Considered Harmful](http://aegis.sourceforge.net/auug97.pdf)
+- [ccache Documentation](https://ccache.dev/manual/latest.html)
+- [Auto-Dependency Generation](https://make.mad-scientist.net/papers/advanced-auto-dependency-generation/)