math-skill 2.0.1 → 2.1.0

package/skills/modeling/original-texts.en.md

@@ -0,0 +1,162 @@
+# Mathematical Sources and Classic Texts
+
+## Newton's *Principia Mathematica* (1687)
+
+> "The same effects of nature must always be assigned to the same causes."
+
+**The most important achievement in the history of mathematical modeling**: Newton unified celestial and terrestrial motion within a single mathematical framework — the law of universal gravitation F = GMm/r² and the three laws of motion. This was humanity's first systematic use of mathematical models to precisely describe the physical world, marking the birth of scientific modeling. The *Principia* established the methodological paradigm of "mathematical model → physical prediction → experimental verification," which remains the foundation of all modeling work to this day.
+
+---
+
+## Fourier's Heat Equation (1822)
+
+> ∂u/∂t = κ ∇²u
+
+In *Théorie analytique de la chaleur*, Fourier proposed a mathematical model for heat diffusion and, in doing so, invented Fourier analysis — the method of decomposing arbitrary functions into trigonometric series. This modeling achievement has a twofold significance: it provided a precise mathematical description of heat conduction, and it gave rise to the entire field of harmonic analysis. The Fourier transform remains a core tool in signal processing, quantum mechanics, image compression, and many other domains.
+
+**Modeling insight**: The new mathematics invented to solve a modeling problem often has a more profound and lasting impact than the original model itself.
+
+---
+
+## Maxwell's Equations (1865)
+
+> ∇·E = ρ/ε₀, ∇×E = -∂B/∂t
+> ∇·B = 0, ∇×B = μ₀J + μ₀ε₀∂E/∂t
+
+Maxwell unified electricity and magnetism into four partial differential equations, and from the mathematical model alone predicted the existence of electromagnetic waves — a prediction later confirmed experimentally by Hertz. This is one of the most brilliant examples in the history of modeling: the discovery of an entirely new physical phenomenon through pure mathematical deduction. The equations also imply the speed of light c = 1/√(μ₀ε₀), subsuming optics into electromagnetic theory — arguably the pinnacle of unification in modeling.
+
+---
+
+## Lotka-Volterra Predator-Prey Model (1925-1926)
+
+The classical population dynamics model:
+
+> dx/dt = αx - βxy (prey growth - predation)
+> dy/dt = δxy - γy (predator growth - natural death)
+
+**Modeling insight**: Through two simple differential equations, this model successfully explains the periodic oscillations observed in predator and prey populations in nature. It is the epitome of a "good model" — simple yet useful.
+
+---
+
+## Kermack-McKendrick SIR Epidemic Model (1927)
+
+> dS/dt = -βSI (decrease in susceptibles)
+> dI/dt = βSI - γI (change in infected)
+> dR/dt = γI (increase in recovered)
+
+**Modeling insight**: The basic reproduction number R₀ = β/γ determines whether an epidemic will take off. Kermack and McKendrick first proposed this model in their 1927 paper "A Contribution to the Mathematical Theory of Epidemics," laying the theoretical foundation for mathematical modeling of infectious diseases. This simple model was widely deployed during the COVID-19 pandemic, demonstrating the enduring value of classical models.
+
+---
+
+## Buckingham Pi Theorem (1914)
+
+> If a physical relationship involves n variables with k independent dimensions, the relationship can be reduced to one among (n-k) dimensionless Π groups.
+
+**Modeling insight**: The Buckingham Pi theorem formalizes dimensional analysis, an extremely important simplification tool in modeling. It tells us that any physical model can be cast in dimensionless form, thereby reducing the number of parameters and revealing essential structure. For example, the Reynolds number Re = ρvL/μ in fluid mechanics is a single Π quantity that unifies countless seemingly different flow regimes.
+
+---
+
+## Turing's Reaction-Diffusion Model (1952)
+
+> ∂u/∂t = D_u ∇²u + f(u,v)
+> ∂v/∂t = D_v ∇²v + g(u,v)
+
+In his paper "The Chemical Basis of Morphogenesis," Turing proved that two chemical substances diffusing at different rates and interacting can generate stable spatial patterns — spots, stripes, spirals — from an initially uniform state. This is a mathematical model of morphogenesis, explaining a wide range of phenomena from leopard spots to seashell patterns.
+
+**Modeling insight**: Mathematical models can explain "how order emerges from disorder" — no pre-existing pattern is required; patterns arise spontaneously from the dynamics of the equations alone. Turing thereby founded the theory of pattern formation.
+
+---
+
+## Lorenz System (1963)
+
+> dx/dt = σ(y - x)
+> dy/dt = x(ρ - y) - xz
+> dz/dt = xy - βz
+
+While numerically simulating an atmospheric convection model, Lorenz discovered that tiny differences in initial conditions lead to completely different long-term behavior — the "butterfly effect." Three seemingly simple equations revealed the essential feature of chaos: the long-term unpredictability of deterministic systems.
+
+**Modeling insight**: Chaos theory profoundly changed the philosophy of modeling — even if a model is perfectly correct, long-term prediction may be fundamentally impossible. Modelers must distinguish between "predictable timescales" and "unpredictable chaotic regimes."
+
+---
+
+## Kalman Filtering (1960)
+
+> x̂_{k|k} = x̂_{k|k-1} + K_k(z_k - H x̂_{k|k-1})
+
+The Kalman filter is a recursive estimation method based on state-space models: it uses the system dynamics model to predict the state and then corrects the prediction with observational data. It unifies "model prediction" and "data updating" within a single mathematical framework and is a cornerstone of modern control theory and signal processing.
+
+**Modeling insight**: Good modeling is not only about "building a model" but also about "making optimal estimates based on the model." The Kalman filter demonstrates how models and data work together — the model provides the prior, the data provide the correction, and neither is dispensable.
+
+---
+
+## Pólya's *How to Solve It* (1945)
+
+> "The first step in mathematical modeling is understanding the problem, the second is devising a plan, the third is carrying out the plan, and the fourth is looking back."
+
+Pólya's problem-solving framework is a precursor to modeling thinking: transforming an unfamiliar problem into a known mathematical problem.
+
+---
+
+## Akaike Information Criterion (1974)
+
+> AIC = -2 ln(L_max) + 2k
+
+In 1974, Akaike proposed AIC, formalizing the model selection problem for the first time: striking an optimal balance between goodness of fit (-2 ln L) and model complexity (2k), where k is the number of parameters and L_max is the maximum likelihood.
+
+**Modeling insight**: AIC established a mathematical standard for the "principle of parsimony" — neither the most complex nor the simplest model is best; rather, the optimal trade-off lies in minimizing information loss while keeping parameters parsimonious. This is the quantitative version of Box's dictum that "some are useful."
+
+---
+
+## Black-Scholes Model (1973)
+
+> C = S N(d₁) - K e^{-rT} N(d₂)
+> d₁ = [ln(S/K) + (r + σ²/2)T] / (σ√T)
+> d₂ = d₁ - σ√T
+
+Black, Scholes, and Merton developed a mathematical model for option pricing. Based on geometric Brownian motion dS = μS dt + σS dW and the no-arbitrage principle, they derived the partial differential equation ∂C/∂t + ½σ²S²∂²C/∂S² + rS∂C/∂S - rC = 0. This model is the most celebrated result in financial mathematics; Scholes and Merton were awarded the 1997 Nobel Prize in Economics for this work.
+
+**Modeling insight**: The Black-Scholes model is the best illustration of Box's dictum — its assumptions (constant volatility, continuous trading, frictionless markets) are all "wrong" in reality, yet it provides the core framework for pricing and risk management, and is therefore "useful."
+
+---
+
+## George Box's Famous Quote
+
+> "All models are wrong, but some are useful."
+
+**Modeling philosophy**:
+- A model is not reality — it is necessarily a simplification of reality
+- The value of a model lies not in its "truthfulness" but in its predictive and explanatory power
+- Criteria for a good model: parsimonious, testable, predictive
+
+---
+
+## General Principles of Modeling
+
+1. **Start simple**: Begin with the simplest model, then incrementally add complexity (Lorenz revealed chaos with three equations; Lotka-Volterra explained oscillations with two)
+2. **State assumptions explicitly**: Record and test every assumption (Black-Scholes assumptions, though wrong, are explicit — and so the model remains usable)
+3. **Validate and falsify**: Test models with independent data, not just by fitting (AIC quantifies the risk of overfitting)
+4. **Know the scope**: Every model has a domain of validity; beyond it, the model fails (Newtonian mechanics fails at high velocities and requires Einstein's correction)
+5. **Iterate**: Modeling is a cyclical process, not a one-shot endeavor (Pólya's "looking back" step)
+6. **Unify dimensions**: Use the Buckingham Pi theorem to cast models in dimensionless form, reducing parameters and revealing structure
+7. **Synergize models and data**: The Kalman filter demonstrates how model prediction and data correction work in a feedback loop
+
+---
+
+## Timeline of Mathematical Modeling
+
+| Year | Achievement | Field |
+|------|-------------|-------|
+| 1687 | Newton *Principia* | Mechanics |
+| 1822 | Fourier heat equation | Heat diffusion |
+| 1865 | Maxwell's equations | Electromagnetism |
+| 1914 | Buckingham Pi theorem | Dimensional analysis |
+| 1925-26 | Lotka-Volterra | Population dynamics |
+| 1927 | Kermack-McKendrick SIR | Epidemiology |
+| 1945 | Pólya *How to Solve It* | Methodology |
+| 1952 | Turing reaction-diffusion | Morphogenesis |
+| 1960 | Kalman filter | Estimation & control |
+| 1963 | Lorenz system | Chaos theory |
+| 1973 | Black-Scholes | Finance |
+| 1974 | Akaike AIC | Model selection |
+
+This timeline reveals a central pattern: great modeling achievements often transcend disciplinary boundaries. Newton unified the heavens and the earth, Maxwell unified electricity and magnetism, Turing unified chemistry and biology — the power of mathematical models lies precisely in their cross-domain universality.

package/skills/optimization/SKILL.en.md

@@ -0,0 +1,129 @@
+---
+name: optimization
+description: |
+  Trigger when a problem involves resource allocation, trade-offs, maximizing/minimizing objectives, decisions under constraints; or needs convexity analysis, Lagrangian/KKT methods, duality structure; or choosing optimization methods for algorithm/operator/training design.
+---
+
+# Optimization
+
+> "Under the most general constraints, find extrema of the objective -- convexity determines difficulty, KKT gives necessity, duality reveals structure."
+>
+> -- Optimization Theory & Operations Research
+
+## Core Principle
+
+**Any decision problem can be formulated as an optimization problem: maximizing (or minimizing) an objective subject to constraints. The essence of optimization is not the pursuit of "the best" in the abstract, but "the best among the feasible."**
+
+The three core elements of optimization: **Objective**, **Constraints**, **Feasible set**.
+
+> **Mathematical Formalization**
+>
+> General optimization problem: $\min_{x \in \mathbb{R}^n} f(x) \quad \text{s.t.} \quad g_i(x) \leq 0,\; i=1,\dots,m; \quad h_j(x) = 0,\; j=1,\dots,p$
+>
+> Lagrangian: $L(x, \lambda, \mu) = f(x) + \sum_i \lambda_i g_i(x) + \sum_j \mu_j h_j(x)$
+>
+> KKT conditions (essential necessary conditions, under Slater-type constraint qualifications): (1) Stationarity $\nabla_x L = 0$; (2) Primal feasibility $g_i \le 0, h_j = 0$; (3) Dual feasibility $\lambda_i \ge 0$; (4) Complementary slackness $\lambda_i g_i = 0$.
+>
+> Convexity: If $f$ and each $g_i$ are convex and each $h_j$ is linear, the problem is convex; in this case **KKT is sufficient**, and local optimality implies global optimality.
+
+## GPU-Friendliness (Cross-Cutting Check)
+
+When optimization is used for **algorithm/operator/training design**, the solution method itself must pass the eight-dimensional gate in `../../references/gpu-friendly-math.md`:
+
+- **First-order methods (SGD/Adam)**: GEMM-friendly, parallelizable, viable in low precision; pay attention to optimizer state precision and distributed communication overhead.
+- **Second-order / Newton methods**: Hessian inversion $O(n^3)$, memory explosion -- the classic "beautiful but incomputable" case -- adapt to **K-FAC / low-rank / diagonal approximations** (see `../../references/books/optimization-ml.md`, `matrix-analysis.md`).
+- **Constraint projection**: Does the projection admit a closed form and is it tensorizable? Iterative projection requires caution regarding serial dependencies.
+- **Distribution**: Can computation and communication overlap? Is gradient compression needed?
+
+Eight-dimensional minimum assessment (formal terms): **Tensorization** -- whether objectives / constraints / gradients can be batched; **GEMM-mappability** -- whether the primary computation is matrix multiplication, HVP, or small K-FAC matrices; **Complexity** -- order of first-order / second-order / combinatorial solvers; **Memory & KV-Cache** -- optimizer states, Hessian storage, activation retention; **Low-precision stability** -- condition numbers, damping, loss scaling; **Parallelism & communication** -- gradient synchronization and communication overlap; **Sparse structure** -- whether preconditioners / constraints are block-structured; **Operator fusion** -- whether updates, clipping, and regularization can be fused.
+
+> Cross-reference `../../references/books/optimization-ml.md` (Chong/Lu/Żak) and `../../references/books/matrix-analysis.md`.
+
+## When NOT to Use
+
+- **No clear evaluation criterion** (one does not know what "good" means) -- define the objective before optimizing.
+- **Purely execution-oriented tasks** (e.g., formatting code) -- there is no optimization space.
+- **The user has already decided on a plan** -- optimization is already complete.
+- **The problem is essentially qualitative judgment rather than quantitative extremum** -- model first, then optimize.
+
+## When to Use
+
+- When one needs to determine whether a problem is convex in order to assess solution difficulty.
+- Choosing optimization methods for algorithm/operator/training design and evaluating their GPU viability.
+- Making rational decisions with quantifiable objectives under constraints.
+- Systematic optimization of experimental design, resource allocation, and hyperparameter / architecture search.
+- When one is uncertain whether the current strategy is optimal and wishes to perform a systematic analysis (convexity, duality, sensitivity).
+
+## Method
+
+### Step 1: Define the Objective
+Clarify what is to be maximized / minimized. Key questions: Single-objective or multi-objective? Is it quantifiable (otherwise, find proxy variables)? Static or dynamic? **Is $f$ convex** (convex implies local = global; non-convex requires vigilance against local extrema)? A wrong objective leads further astray the further one proceeds.
+
+### Step 2: List the Constraints
+Distinguish **hard constraints** (physical / budget / deadline) from **soft constraints** (preferences / quality floors); mathematically classify: inequality $g_i(x)\le 0$ (defines the boundary of the feasible set), equality $h_j(x)=0$ (reduces dimensionality), linear (feasible set is a convex polyhedron) vs. nonlinear (may be non-convex).
+
+### Step 3: Classify the Problem Type
+
+| Type | Objective | Constraints | Key Property | Typical Method |
+|------|-----------|-------------|--------------|----------------|
+| LP (Linear Programming) | Linear | Linear inequalities | Optimum at a vertex | Simplex method |
+| QP (Quadratic Programming) | Quadratic | Linear | Positive-definite QP is convex | Interior-point method |
+| Convex optimization | Convex | Convex inequalities + linear equalities | Local = global | Gradient descent, interior-point methods |
+| Non-convex optimization | Non-convex | Arbitrary | Multiple local extrema | Global search, simulated annealing |
+| Combinatorial optimization | Discrete domain | Arbitrary | Frequently NP-hard | Branch-and-bound, heuristics |
+| Stochastic optimization | Contains random terms | May include stochastic constraints | Expected optimum vs. stochastic feasibility | SAA, robust optimization |
+
+### Step 4: Find the Optimal Solution
+- **LP/QP/Convex**: Exploit convexity; gradient-based or interior-point methods guarantee convergence to the global optimum.
+- **Non-convex**: Multi-start strategies, global search, or relaxation to convex approximations.
+- **Combinatorial**: Exact solutions are often NP-hard -- branch-and-bound for small scale, heuristics / approximations for large scale.
+- **Stochastic**: Sample Average Approximation (SAA) converts to a deterministic approximation.
+- **Insufficient information**: A satisficing solution suffices.
+
+### Step 5: Sensitivity Analysis
+The Lagrange multiplier $\lambda_i^*$ is the **shadow price** of the $i$-th constraint -- relaxing the constraint by one unit improves the objective by approximately $\lambda_i^*$. Complementary slackness: $\lambda_i^*=0$ indicates an inactive constraint (no effect on the optimal solution); $\lambda_i^*>0$ indicates an active constraint (the optimal solution lies precisely at its boundary). Focus on: how the optimal solution changes under small perturbations of constraints / objective, and which constraints are active.
+
+### Step 6: Multi-Objective & Pareto
+Multi-objective problems $f_1,\dots,f_k$ generally have no single optimal solution. Pareto optimality: no feasible solution exists that improves all objectives simultaneously. Methods: **Weighted sum** $\min\sum w_i f_i$ (different weights trace different points on the Pareto front); **$\epsilon$-constraint method** $\min f_1$ s.t. $f_i\le\epsilon_i$ (sweep $\epsilon_i$ to cover the front).
+
+### Step 7: Monitor Constraint Changes
+The optimal solution depends on the constraints -- when constraints change, re-optimization is required. Changes in active constraints have the greatest impact (high shadow prices); small changes in inactive constraints typically do not affect the optimal solution.
+
+## Common Errors
+
+| Error | Critique | Correct Approach |
+|-------|----------|------------------|
+| Optimizing without a clear objective | Direction is undefined | Precisely define the objective first |
+| Ignoring implicit constraints | The "optimal solution" is actually infeasible | Exhaustively verify all constraints |
+| Getting trapped in local optima | Greedy methods on non-convex problems do not guarantee global optimality | Verify convexity; use multi-start / global methods for non-convex problems |
+| Treating the optimum as unique | The optimal solution may not be unique | Check for the existence of multiple equivalent optima |
+| Using single-objective methods for multi-objective problems | Different objectives require trade-offs | Employ Pareto analysis |
+| Failing to verify convexity | Applying convex methods to non-convex problems | Determine convexity before selecting a method |
+| Ignoring duality theory | The dual problem may be easier to solve | Construct the dual and exploit strong duality |
+| Confusing feasibility with optimality | Feasible does not imply optimal | Verify feasibility first, then verify optimality |
+| Ignoring computational / GPU complexity | Second-order methods / combinatorial optimization may be incomputable | Assess complexity, pass the GPU eight-dimensional gate, approximate when necessary |
+| Forgetting to re-optimize | Failing to update when constraints change | Periodically check for constraint changes |
+
+## Operating Procedure
+
+When this skill is triggered, the output must include:
+
+1. **Objective function**: `[Objective]: [Description]` + `[Convexity]: [Convex / Non-convex / Unknown]`
+2. **Constraint list**: Each labeled `[Hard / Soft]` and `[Inequality / Equality]` `[Linear / Nonlinear]`
+3. **Problem type classification**: `[Type]: [LP / QP / Convex / Non-convex / Combinatorial / Stochastic]`
+4. **Feasible set analysis**: Which options are feasible? Which constraints are active?
+5. **Optimal / satisficing solution**: `[Strategy]: [Gradient method / Interior-point / Global search / Satisficing / Pareto]`
+6. **Sensitivity analysis**: Shadow prices of key constraints? How do conclusions change under X% variation?
+7. **GPU viability** (if used for algorithm/operator/training): Does the solution method pass the eight-dimensional gate? Label as friendly / retrofittable / unfriendly, with adaptation recommendations.
+8. **Action recommendations**: Explicitly state "Next, I will..."
+
+**Output must not consist of analysis alone without conclusions.**
+
+## Relations to Other Skills
+
+- **Modeling**: Optimization requires prior modeling -- defining objectives and constraints is itself an act of modeling.
+- **Probability and statistics**: Optimization under uncertainty requires stochastic / robust optimization.
+- **Transformation**: Transforming to the dual problem often makes optimization easier; duality is the most profound transformation in optimization.
+- **Game-theoretic thinking**: Multiple decision-makers simultaneously optimizing constitutes a game; the Nash equilibrium is the stable point of multi-player optimization.
+- **Algorithmic thinking**: Solution methods depend on algorithm design -- convex optimization uses gradient methods, combinatorial optimization requires branch-and-bound / heuristics.
+- **Modern mathematics activation**: `../../references/books/optimization-ml.md` (GPU-friendly optimizers, feasibility of second-order methods), `matrix-analysis.md` (condition numbers, low-rank, preconditioning).

package/skills/optimization/SKILL.md

@@ -5,6 +5,8 @@ description: |
   English: Trigger when a problem involves resource allocation, trade-offs, maximizing/minimizing objectives, decisions under constraints; or needs convexity analysis, Lagrangian/KKT methods, duality structure; or choosing optimization methods for algorithm/operator/training design.
 ---
 
+> **语言路由**：若用户消息为英文，请读取并遵循同目录下的 `SKILL.en.md`，按其操作规程以英文输出；中文消息则继续使用本文件。
+
 # ⚖️ 优化思想 / Optimization
 
 > "在最一般的约束条件下，寻找目标函数的极值——凸性决定难度，KKT 给出必要条件，对偶揭示结构。"

package/skills/optimization/original-texts.en.md

@@ -0,0 +1,167 @@
+# Mathematical Sources and Classic Texts
+
+## Euler-Lagrange Variational Equation & Brachistochrone (1696)
+
+> In 1696, Johann Bernoulli posed the brachistochrone problem: under gravity alone, along which curve does a particle descend from A to B in the shortest time? The answer is the cycloid.
+
+**Core equation**: For the functional J[y] = ∫_{a}^{b} F(x, y, y') dx, the extremizing function satisfies
+
+> ∂F/∂y - d/dx(∂F/∂y') = 0 (Euler-Lagrange equation)
+
+For the brachistochrone, F = √(1 + y'²) / √(2g(y₀ - y)). Substituting into the Euler-Lagrange equation yields the parametric equations of the cycloid: x = a(θ - sin θ), y = a(1 - cos θ).
+
+**Key idea**: The calculus of variations reduces "optimization over an entire class of functions" to solving a differential equation — this is the origin of infinite-dimensional optimization. Euler (1744) systematized the calculus of variations; Lagrange (1788) introduced the δ notation and the method of multipliers, both in the same lineage. The calculus of variations later evolved into optimal control theory (Pontryagin's Maximum Principle, 1961), with wide applications in spacecraft trajectory design, robotic path planning, and many other fields.
+
+## Fermat & Weierstrass Conditions for Unconstrained Optimization
+
+> Fermat (1636): A necessary condition for a differentiable function f to attain a local extremum at x* is ∇f(x*) = 0.
+> Weierstrass (19th c.): If f is twice differentiable and ∇f(x*) = 0, then the Hessian H(x*) is positive definite ⇔ x* is a strict local minimum; H(x*) is negative definite ⇔ x* is a strict local maximum.
+
+For convex functions, ∇f(x*) = 0 is not only necessary but also sufficient — and x* is the global minimum.
+
+**Key idea**: First-order conditions locate stationary points; second-order conditions determine the nature of the extremum — the most fundamental theorem of unconstrained optimization. The condition ∇f = 0 is the convergence target of virtually all iterative optimization algorithms (gradient descent, Newton's method).
+
+## Lagrange Multipliers (1788)
+
+> Finding the extremum of f(x) subject to g(x) = 0 is equivalent to finding the unconstrained extremum of L(x,λ) = f(x) - λg(x).
+
+**Key idea**: Transform a constrained optimization problem into an unconstrained one. At the optimal solution, the gradient of the objective function is parallel to the gradient of the constraint function: ∇f(x*) = λ∇g(x*).
+
+## Pareto Optimality (Vilfredo Pareto, ~1906)
+
+> In multi-objective optimization, a solution x is Pareto optimal if there does not exist y such that f_i(y) ≤ f_i(x) for all i with at least one f_j(y) < f_j(x) (strict improvement).
+
+In *Manuale di economia politica* (1906), Pareto introduced this concept into economics: the efficiency frontier of resource allocation. The Pareto frontier {f(x) : x is Pareto optimal} characterizes all trade-off solutions that cannot be simultaneously improved. Mathematically, for objectives f₁,...,fₘ, the Pareto frontier is {y ∈ ℝᵐ : y = (f₁(x),...,fₘ(x)), x Pareto optimal} — it is a (generally non-convex) surface in objective space.
+
+> Weighted scalarization: Convert multi-objective min f₁,...,fₘ into single-objective min Σw_i f_i (w_i > 0); each weight combination corresponds to a point on the Pareto frontier. This is a standard method for computing the Pareto frontier, but it cannot enumerate all Pareto-optimal solutions on non-convex frontiers.
+
+**Key idea**: Multi-objective optimization has no single "optimal solution" but rather a set of Pareto-optimal solutions among which trade-offs must be made. This is ubiquitous in economics (efficiency vs. equity), engineering (performance vs. cost), and machine learning (accuracy vs. complexity).
+
+## Game Theory & Optimization (von Neumann, 1928; Nash, 1950)
+
+> Nash equilibrium: A strategy profile (s₁*, ..., sₙ*) such that for each player i, s_i* is the best response given the other players' strategies — i.e., no player has an incentive to unilaterally deviate.
+
+Von Neumann (1928) proved the minimax theorem for zero-sum games: max_{x} min_{y} f(x,y) = min_{y} max_{x} f(x,y), which is essentially a precursor of linear programming duality. Nash (1950) generalized the equilibrium concept to non-zero-sum games; a Nash equilibrium is an " mutually optimal" optimization — each player's optimization problem takes the other players' strategies as constraints.
+
+> Existence of Nash equilibrium: Nash (1950) used the Brouwer fixed-point theorem to prove that every finite game has at least one mixed-strategy Nash equilibrium. This reveals a deep connection between game theory and topology.
+
+**Key idea**: Game theory extends optimization from "single-agent decision-making" to "multi-agent interactive decision-making" — optimality is no longer with respect to nature, but the best response to other rational agents.
+
+## Karush-Kuhn-Tucker Conditions (1939/1951)
+
+Necessary conditions for inequality-constrained optimization:
+
+> For min f(x) s.t. g_i(x) ≤ 0, h_j(x) = 0, the optimal solution x* satisfies:
+> - Stationarity: ∇f(x*) + Σμ_i∇g_i(x*) + Σλ_j∇h_j(x*) = 0
+> - Primal feasibility: g_i(x*) ≤ 0, h_j(x*) = 0
+> - Complementary slackness: μ_i · g_i(x*) = 0, μ_i ≥ 0
+
+Karush (1939, master's thesis) first derived these conditions; Kuhn & Tucker (1951) independently rediscovered and widely disseminated them.
+
+**Key idea**: At the optimal solution, either a constraint is inactive (μ_i = 0) or it is tight (g_i(x*) = 0). Complementary slackness is the cornerstone of duality theory — it ties the KKT conditions to a zero duality gap. When a constraint qualification holds (e.g., LICQ: the gradients of active constraints are linearly independent), the KKT conditions are necessary for optimality.
+
+## Linear Programming Duality (Farkas, 1902; Dantzig-von Neumann, 1947)
+
+> Farkas' Lemma (1902): Ax = b, x ≥ 0 has a solution ⟺ Aᵀy ≥ 0, bᵀy < 0 has no solution — this is the geometric foundation of LP duality.
+
+> Strong duality theorem: If the primal LP min cᵀx s.t. Ax = b, x ≥ 0 has a feasible solution, then the optimal values of the primal and the dual max bᵀy s.t. Aᵀy ≤ c are equal. The components of the dual variable y are called shadow prices — they measure the "marginal value" of each constraint.
+
+**Key idea**: Every LP has a "mirror" problem — the primal views the problem in terms of cost, the dual in terms of value. Zero duality gap is a profound fact: the minimum cost of resources equals the maximum revenue valued at shadow prices.
+
+## Simplex Method (Dantzig, 1947)
+
+The fundamental algorithm for linear programming. It searches for the optimal solution by moving along vertices of the feasible region.
+
+> Klee-Minty cube (1973): There exist LP instances for which the simplex method visits all 2ⁿ vertices — exponential worst-case complexity.
+
+> Interior-Point Methods: Karmarkar (1984) proposed a polynomial-time LP algorithm that converges along the central path through the interior of the feasible region, avoiding vertex enumeration. Modern interior-point methods (e.g., Mehrotra predictor-corrector) perform excellently in both practice and theory.
+
+**Key idea**: The optimal solution of a linear program always lies at a vertex of the feasible region. The simplex method exploits this structure by moving along edges; interior-point methods approach from the inside — the two are complementary.
+
+## Duality Theory
+
+> Weak duality: For any optimization problem, the dual optimal value d* ≤ primal optimal value p* (duality gap = p* - d* ≥ 0).
+
+> Strong duality (Slater's condition): If a convex optimization problem has a strictly feasible point (Slater point: g_i(x) < 0 strictly), then d* = p*, and the duality gap is zero.
+
+> Duality provides lower bounds (the primal optimal value ≥ the dual value), which is a critical tool in branch-and-bound, cutting-plane, and related algorithms.
+
+**Key idea**: Duality connects "minimizing cost" with "maximizing value" — two perspectives on the same problem. Slater's condition guarantees "no information loss" in convex optimization.
+
+## Gradient Descent (Cauchy, 1847; Robbins-Monro, 1951)
+
+> Cauchy (1847) proposed the method of steepest descent: x_{k+1} = x_k - α_k ∇f(x_k), iterating along the negative gradient direction with step size α_k. For L-smooth convex functions (Lipschitz-continuous gradient), with step size α = 1/L, f(x_k) - f(x*) ≤ L‖x₀ - x*‖² / (2k) — a convergence rate of O(1/k).
+
+> Newton's Method: x_{k+1} = x_k - H(x_k)⁻¹ ∇f(x_k), using second-order information, achieves quadratic convergence O(‖x_k - x*‖²) near the optimum.
+
+> Stochastic Gradient Descent (SGD): Robbins & Monro (1951) proposed stochastic approximation, replacing the true gradient with a noisy estimate g_k ≈ ∇f(x_k). The conditions Σα_k = ∞, Σα_k² < ∞ guarantee convergence. SGD converges in expectation at the cost of variance — minibatching is a trade-off between variance and efficiency.
+
+**Key idea**: Gradient descent is the most naive form of optimization — "go where it is steepest." SGD extends this idea to large-data settings: one need not examine all the data, only a random sample suffices to approximate the direction. The training of deep neural networks relies almost entirely on SGD and its variants (Adam, AdaGrad, etc.).
+
+## Integer Programming & NP-Hardness (Cook, 1971; Karp, 1972)
+
+> Integer Linear Programming (ILP): min cᵀx s.t. Ax ≤ b, x ∈ ℤⁿ — variables take integer values. Even 0-1 ILP (x ∈ {0,1}ⁿ) is NP-hard.
+
+> Cook (1971) proved that SAT is NP-complete (the Cook-Levin theorem); Karp (1972) listed 21 NP-complete problems, including integer programming, the knapsack problem, and the traveling salesman problem. ILP is NP-hard — no polynomial-time exact algorithm exists (under the assumption P ≠ NP).
+
+> Convex relaxation: Relaxing x ∈ ℤⁿ to x ∈ ℝⁿ yields an LP whose optimal value provides a lower bound for the ILP — this is precisely the core mechanism of branch-and-bound.
+
+**Key idea**: Discrete optimization is fundamentally harder than continuous optimization. The seemingly minor constraint of "integrality" pushes a problem from polynomial-time solvability to NP-hardness. Branch-and-bound and cutting-plane methods are the principal exact methods; heuristics are more commonly used in practice.
+
+## Convex Optimization (Boyd & Vandenberghe, 2004)
+
+> Convex optimization: min f(x) s.t. g_i(x) ≤ 0 (convex), h_j(x) = 0 (affine), x ∈ C (convex set). When f is convex, a local minimum is a global minimum.
+
+> Key properties: (1) Local minimum = global minimum; (2) The feasible region is a convex set; (3) Strong duality holds under Slater's condition; (4) Interior-point methods can solve it in polynomial time.
+
+Important subclasses encompassed by convex optimization:
+- LP (Linear Programming): f linear, constraints linear
+- QP (Quadratic Programming): f quadratic, constraints linear
+- SOCP (Second-Order Cone Programming): constraints include second-order cones ‖Ax + b‖ ≤ cᵀx + d
+- SDP (Semidefinite Programming): constraints include positive semidefinite matrix conditions X ≥ 0
+- GP (Geometric Programming): convex after logarithmic transformation
+
+Boyd & Vandenberghe's *Convex Optimization* (2004) unified all of the above problem classes and became the standard textbook and engineering reference for modern optimization. Convex optimization marks the boundary of "efficiently solvable optimization" — convexity guarantees algorithmic convergence and global optimality of solutions.
+
+**Key idea**: Convex optimization is the "sweet spot" of optimization theory — rich enough to encompass a vast number of practical problems, yet structured enough to guarantee tractability. Non-convex problems are often approximated via convex relaxation.
+
+## Bellman's Principle of Optimality (1957)
+
+> "An optimal policy has the property that whatever the initial state and initial decision are, the remaining decisions must constitute an optimal policy with regard to the state resulting from the first decision."
+
+> Recursive equation: V(s) = max_a { R(s,a) + γ · V(s') }, where s' is the result of the state transition.
+
+**Key idea**: The core of dynamic programming — substructures of an optimal solution are themselves optimal. The Bellman equation decomposes multi-stage decision-making into single-stage subproblems and is the theoretical foundation of reinforcement learning.
+
+## The Philosophical Significance of Optimization
+
+The core of optimization thinking is **making the best choice under constraints** — this is not only a mathematical question but also a question about life. Most decisions in life can be formulated as optimization problems:
+
+- **Objective function**: What do you want to maximize? (Happiness? Achievement? Freedom?)
+- **Constraints**: What limitations do you face? (Time? Money? Ability?)
+- **Feasible region**: What choices are available to you under the constraints?
+- **Optimal solution**: Among all choices, which one maximizes the objective?
+- **Dual perspective**: Viewing the problem in terms of "cost" or "value" yields the same answer — this is the philosophy of strong duality.
+- **Convex vs. non-convex**: Convex optimization has a unique global optimum; non-convex optimization has multiple local optima — the predicament of life lies precisely in the fact that the world is non-convex.
+
+## Timeline of Optimization
+
+| Year | Event |
+|------|-------|
+| 1636 | Fermat proposes the necessary condition ∇f = 0 for unconstrained extrema |
+| 1696 | Johann Bernoulli poses the brachistochrone problem; the seeds of the calculus of variations |
+| 1744 | Euler systematizes the calculus of variations |
+| 1788 | Lagrange publishes the method of multipliers in *Mécanique analytique* |
+| 1902 | Farkas' Lemma — the geometric foundation of LP duality |
+| ~1906 | Pareto introduces the efficiency frontier concept into economics |
+| 1928 | Von Neumann proves the minimax theorem (zero-sum games) |
+| 1939 | Karush derives the KKT conditions (master's thesis) |
+| 1947 | Dantzig invents the simplex method; von Neumann establishes LP duality |
+| 1950 | Nash proves the existence of equilibria in non-zero-sum games |
+| 1951 | Kuhn & Tucker publish the KKT conditions; Robbins-Monro propose stochastic approximation |
+| 1957 | Bellman publishes dynamic programming and the principle of optimality |
+| 1971 | Cook proves NP-completeness (the Cook-Levin theorem) |
+| 1972 | Karp lists 21 NP-complete problems |
+| 1973 | Klee-Minty prove exponential worst-case complexity of the simplex method |
+| 1984 | Karmarkar proposes the polynomial-time interior-point method |
+| 2004 | Boyd & Vandenberghe publish *Convex Optimization* |