Introduction

Every living being is, at every moment, doing something — eating, resting, talking, travelling, working, investing. Behind each of these acts lies a choice, and behind each choice lies the silent presumption that the chooser had a reason to prefer it to the alternatives. Optimization is the mathematics of that presumption: it studies how an agent who can rank the available courses of action picks a best one. This book is about the dynamic case — when the act is not once-and-for-all but stretches over time, when today’s choice reshapes tomorrow’s options, and when “best” must be reckoned over a whole path rather than a single instant.

This introductory chapter does three things. First (The origin of optimization problems) it traces where an optimization problem comes from — from the notion of rationality, through an order on the set of choices, to a numerical objective whose maximization encodes that order. Second (Classification of optimization problems and Mathematical statement) it classifies optimization problems and fixes the standard mathematical statement, culminating in the three canonical dynamic problems — the calculus of variations, optimal control, and dynamic programming — together with their central tools and the bridge that unifies them. Third (A few examples) it sets up a handful of motivating examples — the knapsack, consumer choice, resource extraction, cake-eating, and the brachistochrone — as optimization problems, deferring their solutions to the method chapters they motivate.

This is a framing chapter. Its job is to orient, not to prove; the rigour begins in earnest in Static Optimization and never lets up thereafter.

The origin of optimization problems

Rationality

What determines a person’s behaviour? The factors split into the external (coercion, example, the pull of the crowd) and the internal (instinct, habit, experience, character, belief, knowledge, taste). Among the internal factors one — rationality — is the one economics builds on, and the one that makes optimization possible. We adopt a deliberately narrow working definition: rationality is the capacity for logical inference and precise computation; it is a mode of thought that prizes rigorous deduction over intuition, hearsay, or the worship of forces one does not understand. In this sense rationality is, as von Mises argued at length in Human Action (Mises 1949), the defining feature of deliberate human conduct, and one of the twin pillars of modern science (the other being the discipline of experiment).

It helps to distinguish two grades of rationality.

Complete versus bounded rationality. An agent has complete rationality if, facing a situation, they can survey every feasible course of action and select the most advantageous one and carry it out. An agent has bounded rationality if they want to choose the most advantageous action but their powers of inference and computation fall short of identifying it. Real agents are boundedly rational; the theory of optimization idealizes them as completely rational and then studies how far the idealization can be pushed.

Two words in that definition deserve scrutiny: advantageous and most.

  • The “advantage” (the gain, the good) need not be selfish. Under a given moral framework, every action has a well-defined worth, and that worth may incorporate altruism — concern for others’ welfare — without ceasing to be a sharp, quantifiable standard. A purely self-regarding agent is the special case in which “advantage” means private benefit alone. So long as the worth of each action is unambiguous, the rational-agent method applies, even to the analysis of moral behaviour. In physics and engineering the “advantage” is sharper still and carries no ethical content: there the subject of the problem recedes — nature itself is the optimizer, and physics keeps discovering how relentlessly nature extremizes.

  • The word “most” is the deep one: it presupposes that all the feasible actions have been arranged in an order, so that a top element can be spoken of. Making that order precise is the real content of setting up an optimization problem, and it occupies the rest of this section.

The five steps of a rational agent

Faced with a matter to settle, a completely rational agent proceeds through five steps:

  1. Frame the matter — state clearly what is to be decided.
  2. Enumerate all courses of action available to the agent; collect them into a set, the choice set \(X\).
  3. Order \(X\) — install on it a relation \(\preceq\), read “is no better than” (the course writes this as \(\le\) and reads it “is inferior to”).
  4. Optimize — find a \(\preceq\)-greatest element of \(X\).
  5. Execute it.

If step 1 fails — if the matter cannot even be framed — nothing else can begin; and the way a matter is framed strongly conditions everything downstream. To model a real situation is precisely to turn a vague “matter” into a definite “problem”, i.e. to perform steps 1–3 explicitly. This book concentrates almost entirely on step 4: given the problem, find the optimum — and chiefly the exact (analytic) optimum, or, where no closed form exists, the qualitative properties of the optimum (existence, uniqueness, monotonicity, asymptotic behaviour). We do not pursue numerical algorithms for approximate solutions.

An order on the choice set

The heart of the construction is the order installed in step 3. We make it precise.

Definition 1 (Preorder (the course’s “order”)) A binary relation \(\preceq\) on a set \(X\) is a preorder (the course simply calls it an order) if it is

  • reflexive: \(x\preceq x\) for every \(x\in X\);
  • transitive: if \(x_1\preceq x_2\) and \(x_2\preceq x_3\) then \(x_1\preceq x_3\).

We read \(x_1\preceq x_2\) as “\(x_1\) is no better than \(x_2\)” (equivalently \(x_2\succeq x_1\), “\(x_2\) is at least as good as \(x_1\)”). If \(x\preceq y\) and \(y\preceq x\) we say \(x\) and \(y\) are indifferent, written \(x\sim y\). If \(x\preceq y\) but not \(y\preceq x\) we write \(x\prec y\) and say \(x\) is strictly inferior to \(y\).

On the symbol \(\preceq\) versus \(\le\). The course writes the abstract order with the ordinary inequality sign \(\le\) on the choice set \(\aleph\). To avoid collision with the numerical \(\le\) on \(\mathbb{R}\) — which appears the moment we introduce a real-valued objective — we write the abstract order as \(\preceq\) and reserve \(\le\) for numbers. The choice set itself we write \(X\), not \(\aleph\) (see notation).

Definition 2 (Optimal element) An element \(x^\ast\in X\) is optimal for \(\preceq\) if no element strictly dominates it: there is no \(x\in X\) with \(x^\ast\prec x\). Equivalently, \(x^\ast\) is a \(\preceq\)-maximal element of \(X\).

Reflexivity and transitivity are exactly the structure needed to make “no better than” behave like a comparison, but they stop short of guaranteeing that any two elements can be compared at all. That gap is the source of two genuinely different regimes.

The Pareto order and incomparability

A preorder under which some pairs are incomparable — neither \(x\preceq y\) nor \(y\preceq x\) — is called a partial order, and in economics the most important partial order is the Pareto order. It arises whenever a choice carries several incommensurable dimensions of value. The classic construction: each course of action \(x\) is scored by a vector of attributes \((x^1,\dots,x^n)\in\mathbb{R}^n\), and we declare

\[ (x^1,\dots,x^n)\ \preceq_{\mathrm{P}}\ (y^1,\dots,y^n) \quad\Longleftrightarrow\quad x^i\le y^i\ \text{ for every } i=1,\dots,n . \tag{1}\]

This is the product order: a course is Pareto-better only if it is at least as good on every attribute. Two courses that each beat the other on some attribute are incomparable. A \(\preceq_{\mathrm{P}}\)-optimal element — one that cannot be improved in any dimension without being worsened in another — is called Pareto optimal. A Pareto optimum need not exist, and when it exists it need not be unique.

Example 1 (Choosing among incomparable options) Suppose one must rank a set \(X\) of candidates, scoring each by character and competence, \(f(x)=(f^1(x),f^2(x))\in\mathbb{R}^2\). The natural order on \(\mathbb{R}^2\) is the Pareto order 1: candidate \(x\) is unambiguously worse than \(y\) only if \(y\) is at least as good in both character and competence. Two candidates, one of finer character and the other of greater competence, are incomparable under \(\preceq_{\mathrm{P}}\). Since one must finally pick a single candidate, an incomplete order will not do; one is forced to complete it — typically by a weighted sum \(\theta_1 f^1(x)+\theta_2 f^2(x)\) that trades the two attributes off against each other at fixed rates \(\theta_1,\theta_2>0\). Whenever the attributes are commensurable, this collapses the partial order to a complete one and a best element reappears.

Preferences: a complete order

Adjoin to a preorder one more axiom and the incomparable pairs disappear.

Definition 3 (Complete order (preference)) A preorder \(\preceq\) on \(X\) is a complete order (a total order, an economic preference) if it is

  • complete: for every \(x,y\in X\), \(x\preceq y\) or \(y\preceq x\) (every pair is comparable).

Under a complete order any two courses of action can be ranked, so — on a finite or suitably well-behaved \(X\) — a best element exists.

The methodological counsel that follows is simple: complete the order whenever you can. A complete order is what makes the phrase “the most advantageous action” meaningful and the optimization problem well posed. Sometimes, though, completion is genuinely impossible — when each attribute has an irreducible meaning that admits no rate of substitution against the others, no weighted sum is legitimate, and one must rest content with the Pareto order and its Pareto optima.

From the order to a numerical objective

How does one install a complete order on \(X\) in practice? Almost always by evaluating the options — assigning each a value and inheriting the order of the values. Formally, one chooses a value set \(Y\) that already carries a natural order \(\le_Y\), and a map

\[ f : X \longrightarrow Y , \tag{2}\]

the objective, which pulls back the order on \(Y\) to an order on \(X\):

\[ x_1 \preceq x_2 \quad\Longleftrightarrow\quad f(x_1)\le_Y f(x_2). \tag{3}\]

The map induces the order. Equation Equation 3 is the workhorse of the whole subject. Comparing courses of action directly is hard; comparing the numbers (or vectors) they produce is easy, because \(Y\) comes pre-ordered. The art of modelling is to find an \(f\) whose order Equation 3 faithfully reproduces the agent’s true ranking.

Two cases recur:

  • \(Y=\mathbb{R}\) (a single real-valued objective). The order \(\le_Y\) is the usual order on the line — automatically complete — so \(f:X\to\mathbb{R}\) delivers a complete order for free, and “find the best \(x\)” becomes “maximize \(f\)”. This is the case the whole book treats.

  • \(Y=\mathbb{R}^n\), \(n>1\) (a vector-valued objective), with \(\le_Y\) the Pareto order

    1. Now \(f\) induces only a partial order; the problem is multi-objective, and one either accepts Pareto optimality or completes the order by a weighted sum, as in

We will see in Mathematical statement that maximizing \(f\) is the standard form of every optimization problem in this book; the construction just given is why that form captures rational choice.

Classification of optimization problems

The same abstract problem \(\max_{x\in X}f(x)\) wears many guises. Four cross-cutting distinctions organize them; this book lives in particular cells of the resulting grid.

(i) Static versus dynamic — does time enter? If an action is taken once and its consequences for the future may be ignored, the problem is static: one chooses a single point \(x\in X\) and is done. If instead the action’s effect on the future cannot be neglected — or, more sharply, if one must choose a whole sequence or path of actions whose cumulative effect is what matters — the problem is dynamic: the unknown is itself a sequence \((x_0,x_1,\dots)\) or a function of time \(x(\cdot)\). The dividing line is not whether time appears in the formulas but whether the coupling of present action to future consequence is strong enough to matter. “What is good in the short run need not be good in the long run” is the slogan of the dynamic regime, and the reason it demands its own methods.

(ii) Discrete versus continuous time. A dynamic problem is discrete-time when decisions are made at a sequence of isolated instants \(t=0,1,2,\dots\) (the unknown is a sequence), and continuous-time when decisions are made throughout an interval \([0,T]\), possibly with \(T=\infty\) (the unknown is a path \(x:[0,T]\to\mathbb{R}^n\)). The two are not cosmetic variants: the discrete problem is governed by difference equations and recursions, the continuous one by differential equations, and their natural tools differ accordingly.

(iii) Deterministic versus stochastic. When the agent’s environment is known with certainty, the problem is deterministic. When randomness intervenes — the law of motion or the payoff is buffeted by chance — the problem is stochastic, the objective becomes an expected value, and the tools acquire probabilistic content (conditional expectations, Itô calculus).

(iv) By the type of decision variable. When the decision variable ranges over a discrete set the problem is combinatorial, a family that includes integer programming and 0–1 programming (binary choices). When the decision variable ranges over a continuum the problem is the smooth optimization of calculus, and carries no special name.

Where this book sits. The course — and this reconstruction — develops continuous-variable dynamic optimization, mostly deterministic. The calculus of variations and optimal control are treated in continuous time only; dynamic programming is treated in both discrete and continuous time, and is where the stochastic case enters. Static optimization is included as a foundation and a foil (Mathematical statement), and the qualitative theory of ordinary differential equations as the prerequisite for everything continuous-time. Combinatorial problems such as the knapsack (Example 6.4) appear only to motivate; we do not develop their algorithmics.

Mathematical statement

The standard form

Every optimization problem in this book is written in the standard form

\[ \max_{x\in X} f(x), \qquad\text{equivalently}\qquad \max_x\ f(x)\ \ \text{s.t.}\ \ x\in X , \tag{4}\]

where “s.t.” abbreviates subject to. The ingredients have fixed names:

  • \(X\) is the feasible set (the choice set, the admissible decisions);
  • \(x\) is the decision variable (optimization variable);
  • \(f\) is the objective, and \(\max\) means the greatest value of \(f\) under the order on its codomain.

When the codomain of \(f\) is \(Y=\mathbb{R}\) the problem is single-objective; the order is the complete order of the real line and a maximum is a genuine maximum. When \(Y=\mathbb{R}^n\) with \(n>1\) the problem is multi-objective, the order is Pareto, and “max” means a Pareto optimum. This book treats only the single-objective case, \(Y=\mathbb{R}\).

Decision variables versus parameters. Writing the decision variable underneath the \(\max\) is not decoration — it declares which symbols are being chosen and which are held fixed. Every symbol not under the \(\max\) is a parameter: a constant for the purposes of this problem, though one may later ask how the optimum responds to it (comparative statics). The distinction is consequential. Compare \[ \max_{x}\ \theta x-x^2 \quad\text{s.t.}\quad x\in[0,\theta] \qquad\text{versus}\qquad \max_{(x,\theta)}\ \theta x-x^2 \quad\text{s.t.}\quad x\in[0,\theta],\ \theta\in[0,1]. \] In the first, \(\theta\) is a fixed parameter and the optimizer chooses only \(x\) (obtaining a value \(V(\theta)\) that depends on \(\theta\)); in the second, \(\theta\) is also chosen. The two are different problems. They are related, however, by the nested-maximization identity \[ \max_{(x,\theta)}f(x,\theta) =\max_{\theta}\Bigl(\max_{x}f(x,\theta)\Bigr) =\max_{x}\Bigl(\max_{\theta}f(x,\theta)\Bigr), \tag{5}\] which says a joint maximum may be found by optimizing the variables in either order, one block at a time. This iterated-optimization identity is, in embryo, the principle behind dynamic programming.

In the static problems of Chapter 1 the feasible set is a subset of Euclidean space carved out by constraints, \(X=\{x\in\mathbb{R}^n: g(x)\ge 0,\ h(x)=0\}\), and \(f:X\to\mathbb{R}\) is an ordinary function. In the dynamic problems the feasible set is a subset of a function space — paths \(x(\cdot)\) in \(C^1[0,T]\) in continuous time, sequences in continuous time, sequences \((x_t)\) in discrete time — and the objective is a functional: an integral \(\int_0^T(\cdots)\,\mathrm{d}t\) in continuous time or a sum \(\sum_t(\cdots)\) in discrete time. The decision variable is then an entire function or sequence, which is exactly what makes dynamic optimization harder than static: one is choosing not a number but a curve.

The three canonical dynamic problems

The continuous-time dynamic-optimization problems that economics needs are solved by three methods, each with its own standard problem and its own central tool. They form the spine of this book.

Table 1: The three methods of dynamic optimization, their standard problems, and their central tools.
Method Standard problem Central tool
Calculus of variations \(\displaystyle\max\int_0^T F(t,x,\dot x)\,\mathrm{d}t\), \(\ x(0)=x_0\) Euler equation
Optimal control \(\displaystyle\max\int_0^T f(t,x,u)\,\mathrm{d}t\) s.t. \(\dot x=g(t,x,u)\), \(\ x(0)=x_0\) Pontryagin’s maximum principle
Dynamic programming \(\displaystyle\max\sum_{t=0}^{T}\beta^t\,\pi(t,x_t,u_t)\) s.t. \(x_{t+1}=f(t,x_t,u_t)\), \(\ x_0\) given Bellman / HJB equation

Read the table as three answers to one question — “which path is best?” — that differ in what is chosen and how the dynamics are written.

Calculus of variations. The unknown is the path \(x(\cdot)\) itself; its rate of change \(\dot x\) enters the integrand \(F(t,x,\dot x)\) directly. With \(F\) twice differentiable, \(T\) a positive number or \(\infty\), and \(x_0\) a constant, the necessary condition for optimality is the Euler equation \[ F_x = \frac{\mathrm{d}}{\mathrm{d}t}F_{\dot x}, \tag{6}\] a (generally second-order) differential equation for the optimal path. This is the subject of Chapter 3.

Optimal control. Here one separates the state \(x\) (what the system is) from the control \(u\) (the lever the agent pulls), linked by a state equation (law of motion) \(\dot x=g(t,x,u)\); the agent chooses \(u(\cdot)\) and the state responds. The running payoff is \(f(t,x,u)\). The necessary condition is Pontryagin’s maximum principle (Pontryagin et al. 1962), organized around the Hamiltonian \[ H(t,x,u,\lambda)=f(t,x,u)+\lambda\,g(t,x,u), \tag{7}\] in which the multiplier \(\lambda\) on the state equation — the costate — is the shadow price of the state. The maximum principle says the optimal control maximizes \(H\) pointwise while the costate obeys its own differential equation \(\dot\lambda=-H_x\). This is Chapter 4. The calculus of variations is the special case \(g(t,x,u)=u\), \(\dot x=u\) — i.e. the control is the velocity — so optimal control strictly generalizes it.

Dynamic programming. Written here in discrete time, the agent maximizes a discounted sum of per-period payoffs \(\pi(t,x_t,u_t)\) subject to a transition \(x_{t+1}=f(t,x_t,u_t)\), with the discount factor \(\beta=\tfrac{1}{1+\rho}\in(0,1)\). The central tool is the Bellman equation (Bellman 1957), the recursion \[ V_t(x)=\max_{u}\bigl\{\pi(t,x,u)+\beta\,V_{t+1}\bigl(f(t,x,u)\bigr)\bigr\}, \tag{8}\] for the value function \(V\) — the optimized objective seen as a function of the current state. Its continuous-time analogue is the Hamilton–Jacobi–Bellman (HJB) partial differential equation. This is Chapter 5.

The discount trap (flagged once, honoured everywhere). The continuous-time objectives carry a discount rate \(\rho>0\) inside \(e^{-\rho t}\); the discrete-time Bellman objective carries a discount factor \(\beta\in(0,1)\). The two are linked through the rate \(\rho\): per-period compounding gives \(\beta=1/(1+\rho)\), and continuous compounding gives \(\beta=e^{-\rho}\), the two agreeing to first order (\(e^{-\rho}=1-\rho+O(\rho^2)\) and \(1/(1+\rho)=1-\rho+O(\rho^2)\)). We take \(\beta=1/(1+\rho)\) as the canonical factor (per notation). The course overloads the single symbol \(\rho\) for both rate and factor; throughout these notes \(\rho\) is always the continuous rate and \(\beta\) is always the discrete factor.

The unifying bridge

The three methods are three faces of one theory. The optimal-control and dynamic-programming formulations describe the same optimal path, and the link between them is exact and illuminating:

Costate \(=\) gradient of the value function. Pontryagin’s costate \(\lambda(t)\) equals the gradient of Bellman’s value function evaluated along the optimal path, \[ \lambda(t)=V_x\bigl(t,x^\ast(t)\bigr). \tag{9}\] Both quantities are the same shadow price of the state: the marginal value, at the optimum, of an extra unit of the state variable. Read this way, the maximum principle’s costate equation \(\dot\lambda=-H_x\) and the HJB equation are two encodings of the same economics, and the Hamiltonian Equation 7 is the HJB equation’s maximand. We develop Equation 9 carefully once both machines are built; for now it is the promise that the three chapters tell one story.

A second bridge runs to the static theory of Chapter 1. The Hamiltonian and the Bellman maximand are each maximized by a first-order (KKT-type) condition in the control, so at every instant a dynamic problem hides a static one. The costate and the value-function gradient are the dynamic descendants of the static Lagrange multiplier and envelope derivative. This is why we begin with static optimization and the qualitative theory of ODEs before turning to the dynamic machinery: the former supplies the optimization grammar, the latter the language in which continuous-time optima are written.

A few examples

We close by posing — not solving — several problems that recur as test cases throughout the book. Each is stated cleanly in the standard form Equation 4, with a pointer to the chapter where its method is developed. The aim is to populate the abstractions above with concrete decision variables, feasible sets, and objectives.

Static and combinatorial examples

Example 2 (The knapsack problem (combinatorial)) A traveller with a knapsack of capacity \(W\) faces \(n\) stones of weights \(\omega_1,\dots,\omega_n\) and values \(v_1,\dots,v_n\), and must choose which to carry to maximize total value. With a binary decision variable \(x_i\in\{0,1\}\) (carry stone \(i\), or not), the problem is \[ \max_{(x_1,\dots,x_n)}\ \sum_{i=1}^{n} v_i\,x_i \qquad\text{s.t.}\qquad \sum_{i=1}^{n}\omega_i\,x_i\le W,\quad x_i\in\{0,1\}\ \ (i=1,\dots,n). \] This is a 0–1 (binary) program: the decision variable is a vector of bits, the feasible set is combinatorial, and the methods of smooth calculus do not apply directly. We list it to mark the boundary of this book’s scope (continuous variables); its natural home is integer programming and — when stages are introduced — the discrete dynamic programming of Chapter 5.

Example 3 (The consumer’s problem (static, continuous)) An agent with budget \(C>0\) buys quantities \(x_1,\dots,x_n\) of \(n\) goods at prices \(p_1,\dots,p_n>0\) to maximize utility \(U(x_1,\dots,x_n)\): \[ \max_{(x_1,\dots,x_n)}\ U(x_1,\dots,x_n) \qquad\text{s.t.}\qquad p_1 x_1+\cdots+p_n x_n\le C,\quad x_1\ge 0,\dots,x_n\ge 0 . \] The decision variable is a continuous consumption bundle; the feasible set is the budget simplex (a polytope cut out by one inequality constraint and \(n\) non-negativity constraints). This is the prototypical static, constrained, continuous problem, solved by the Kuhn–Tucker conditions of Chapter 1.

Continuous-time dynamic examples

Example 4 (Optimal resource extraction (continuous-time, optimal control)) An owner holds the rights to a mineral deposit for \(T\) years. Let \(x(t)\) be the stock remaining at time \(t\) and \(y(t)\ge 0\) the extraction rate, so the stock depletes according to \(\dot x=-y\). The product sells at price \(p(t)\) in a competitive market; extracting at rate \(y\) from a stock \(x\) costs \(c(x,y)\) per unit time; the discount rate is \(\rho>0\) and the initial stock is \(x(0)=x_0\). The owner solves \[ \max_{y(\cdot)}\ \int_0^T e^{-\rho t}\bigl[\,p(t)\,y - c(x,y)\,\bigr]\,\mathrm{d}t \qquad\text{s.t.}\qquad \dot x=-y,\quad x(0)=x_0,\quad y\ge 0, \] which is a textbook optimal-control problem with state \(x\) (the stock) and control \(y\) (the extraction rate). Eliminating the control via \(y=-\dot x\) turns it into the equivalent variational problem \[ \min_{x(\cdot)}\ \int_0^T e^{-\rho t}\bigl[\,p(t)\,\dot x + c(x,-\dot x)\,\bigr]\,\mathrm{d}t \qquad\text{s.t.}\qquad x(0)=x_0 , \] a concrete instance of the equivalence “calculus of variations \(=\) optimal control with \(\dot x=u\)” noted after Equation 7. Its solution — Hotelling’s rule, that the resource’s marginal value rises at the discount rate — is derived in Chapter 3 and revisited with the maximum principle in Chapter 4.

A notational convention used throughout the dynamic chapters and already visible above: a time-dependent quantity is written \(x\) rather than \(x(t)\) when no confusion results, and a dot denotes the time derivative, \(\dot x=\mathrm{d}x/\mathrm{d}t\).

Discrete-time dynamic example

Example 5 (The cake-eating problem (discrete-time, dynamic programming)) A consumer owns a cake of size \(1\) and, each period \(t=0,1,2,\dots\), eats an amount \(c_t\ge 0\), enjoying instantaneous utility \(u(c_t)\); total utility is the discounted sum of flow utilities, with discount factor \(\beta\in(0,1)\). Writing \(x_t\) for the cake remaining at the start of period \(t\) (so \(x_0=1\) and \(x_{t+1}=x_t-c_t\)), the consumer solves \[ \max_{(c_t)_{t\ge 0}}\ \sum_{t=0}^{\infty}\beta^{t}\,u(c_t) \qquad\text{s.t.}\qquad x_{t+1}=x_t-c_t,\quad 0\le c_t\le x_t,\quad x_0=1 . \] This is the canonical dynamic-programming problem: state \(x_t\) (cake remaining), control \(c_t\) (amount eaten), transition \(x_{t+1}=x_t-c_t\), discounted-sum objective. It is the simplest non-trivial test of the Bellman equation Equation 8 and is solved in Chapter 5.

The course originally writes the discount factor as \(\rho\in(0,1)\); per the discount-trap convention we write it \(\beta\), reserving \(\rho\) for the continuous-time rate.

The brachistochrone: where the subject began

Example 6 (The brachistochrone (the founding problem of the calculus of variations)) In a vertical plane, fix two points \(O\) (the origin) and \(A=(p,q)\) that share neither a horizontal nor a vertical line, with \(O\) above \(A\). Along which frictionless track does a bead, released from rest at \(O\) and pulled by gravity, slide to \(A\) in the shortest time? With the \(y\)-axis pointing downward and the curve written \(y=y(x)\), energy conservation \(\tfrac12 m v^2=m g y\) gives the speed \(v=\sqrt{2gy}\), and since an element of arc \(\mathrm{d}s=\sqrt{1+y'^2}\,\mathrm{d}x\) is traversed in time \(\mathrm{d}t=\mathrm{d}s/v\), the total descent time is the functional \[ T[y]=\int_0^p \frac{\sqrt{1+y'^2}}{\sqrt{2gy}}\,\mathrm{d}x . \] Dropping the constant \(\sqrt{2g}\), the brachistochrone problem is the variational problem \[ \min_{y(\cdot)}\ \int_0^p \sqrt{\frac{1+y'^2}{y}}\,\mathrm{d}x \qquad\text{s.t.}\qquad y(0)=0,\ \ y(p)=q . \] Galileo (1630) conjectured a circular arc; Johann Bernoulli (1696), Jacob Bernoulli, Leibniz, Newton, and l’Hôpital independently found the true answer — a cycloid, with parametric form \(x=a(\theta-\sin\theta)\), \(y=a(1-\cos\theta)\), the curve traced by a point on a rolling wheel. Johann Bernoulli’s ingenious solution treated the descent as light refracting through ever-finer horizontal strata, in each of which the bead moves in a straight line at constant speed; the optimal angle in each stratum obeys Snell’s law \(\sin\alpha/v=\text{const}\), exactly the optical extremal principle of Fermat. Jacob Bernoulli’s solution, by contrast, introduced the method of varying the whole curve — the embryo of the calculus of variations — and led to the same governing differential equation \(y\bigl(1+y'^2\bigr)=A\), whose solution is the cycloid. It was Euler who, building on these solutions, forged the general method and the milestone result that bears his name, the Euler equation Equation 6. We solve the brachistochrone in Chapter 3.

A historical aside. The cycloid is also the tautochrone (isochrone) — the curve along which a bead reaches the lowest point in a time independent of its starting height. That a single curve solves both problems was shown by Abel, who reduced the tautochrone to an integral equation and cracked it with the Laplace transform and the convolution theorem. The brachistochrone thus stands at the crossroads of mechanics, optics, and analysis, and is rightly called the opening problem of dynamic optimization.

These examples make a single methodological point visible. A static optimization problem asks for a best number (or vector); a dynamic problem asks for a best function — a whole path. That is why dynamic optimization is so much harder, and why it needs the dedicated machinery of the chapters ahead. And because every continuous-time method below ultimately reduces optimality to a differential equation — the Euler equation, the costate equation, the HJB equation — the prerequisite for all of them is a working command of ordinary differential equations. We therefore supply, after static optimization, a self-contained chapter on the qualitative theory of ODEs (Chapter 2) before opening the dynamic theory proper.