The Mathematics of Optimization

Whether an airline is scheduling flights, a farmer is dividing land between crops, or a phone is choosing the fastest route home, the underlying question is the same: what is the best possible choice? Optimization is the branch of mathematics that answers this question precisely. In this article we will look at the main tools, from school calculus to the algorithms that train artificial intelligence.

The Language of Optimization

Every optimization problem has three ingredients:

• Decision variables: the quantities you can choose, such as the length of a fence or the number of products to make.
• An objective function: the quantity you want to maximise (profit, area, volume) or minimise (cost, time, error).
• Constraints: the rules your choices must obey, such as a fixed budget or limited raw materials.

Writing a problem in these terms is often half the work. Once it is written down clearly, there is usually a standard technique waiting to solve it.

Calculus: Finding Turning Points

For a smooth function of one variable, the maximum or minimum values inside an interval occur where the graph is flat, that is, where the derivative is zero. The procedure is:

1. Write the objective as a function of one variable.
2. Differentiate and set the derivative equal to zero.
3. Solve for the critical points.
4. Use the second derivative (or a sign check) to decide whether each point is a maximum or minimum.
5. Check the endpoints of the allowed interval too.

If \( f''(x) < 0 \) at a critical point, the curve bends downwards and the point is a local maximum. If \( f''(x) > 0 \), it is a local minimum.

Worked Example 1: Fencing a Field by a River

A farmer has 200 m of fencing to enclose a rectangular field beside a straight river. No fence is needed along the river. What dimensions give the largest area?

Let \( x \) be the length of each side perpendicular to the river. The side parallel to the river then has length \( 200 - 2x \). The area is

$$ A(x) = x(200 - 2x) = 200x - 2x^2 $$

Differentiate:

$$ A'(x) = 200 - 4x $$

Set this equal to zero:

$$ 200 - 4x = 0 \quad \Rightarrow \quad x = 50 $$

Check the second derivative: \( A''(x) = -4 < 0 \), so this is a maximum. The parallel side is \( 200 - 2(50) = 100 \) m, and the maximum area is

$$ A = 50 \times 100 = 5000 \text{ m}^2 $$

Notice the best shape is not a square: because the river provides one side for free, it pays to make the parallel side twice as long.

Worked Example 2: The Open Box

A square sheet of cardboard 30 cm wide has equal squares of side \( x \) cut from each corner, and the sides are folded up to make an open box. Find \( x \) for maximum volume.

The base becomes a square of side \( 30 - 2x \) and the height is \( x \), so

$$ V(x) = x(30 - 2x)^2, \qquad 0 < x < 15 $$

Using the product rule:

$$ V'(x) = (30 - 2x)^2 + x \cdot 2(30 - 2x)(-2) = (30 - 2x)^2 - 4x(30 - 2x) $$

Factor out \( (30 - 2x) \):

$$ V'(x) = (30 - 2x)(30 - 2x - 4x) = (30 - 2x)(30 - 6x) $$

Setting \( V'(x) = 0 \) gives \( x = 15 \) or \( x = 5 \). The value \( x = 15 \) produces a box with zero base, so the maximum is at \( x = 5 \) cm. The volume is

$$ V(5) = 5 \times 20^2 = 2000 \text{ cm}^3 $$

A quick check: \( V(4) = 4 \times 22^2 = 1936 \) and \( V(6) = 6 \times 18^2 = 1944 \), both smaller, confirming the maximum.

Linear Programming

Many business problems have linear objectives and linear constraints. These are solved by linear programming, developed in the 1940s by Leonid Kantorovich in the Soviet Union and George Dantzig in the United States, whose simplex method is still used today.

The key fact is that the optimum always occurs at a corner (vertex) of the feasible region.

Worked Example 3: Maximising Profit

A workshop makes chairs (\( x \)) and tables (\( y \)). Profit, measured in hundreds of rupees, is 40 per chair and 30 per table. There are at most 40 units of wood and 60 hours of labour; a chair uses 1 unit of wood and 2 hours, a table uses 1 unit of wood and 1 hour. Maximise

$$ P = 40x + 30y $$

subject to \( x + y \le 40 \), \( 2x + y \le 60 \), \( x \ge 0 \), \( y \ge 0 \).

Find the corners. The lines \( x + y = 40 \) and \( 2x + y = 60 \) meet where subtracting gives \( x = 20 \), so \( y = 20 \). The other corners are \( (0,0) \), \( (30,0) \) and \( (0,40) \).

Evaluate \( P \) at each corner:

The best plan is 20 chairs and 20 tables, giving a profit of 1400 hundred rupees, or Rs 140,000. Making only one product wastes either wood or labour.

Constrained Optimization and Lagrange Multipliers

When the objective depends on several variables and is tied by a curved constraint, Joseph-Louis Lagrange's method is used. To optimise \( f(x, y) \) subject to \( g(x, y) = c \), we solve

$$ \nabla f = \lambda \nabla g $$

together with the constraint. The multiplier \( \lambda \) has a useful meaning: it tells you how much the optimum improves if the constraint is relaxed slightly. Economists call it a shadow price.

Gradient Descent: Optimization for Computers

When a function has thousands or millions of variables, as in machine learning, we cannot solve the equations directly. Instead, gradient descent takes small steps downhill:

$$ x_{n+1} = x_n - \alpha f'(x_n) $$

where \( \alpha \) is the learning rate. For \( f(x) = (x - 3)^2 \) with \( f'(x) = 2(x - 3) \), starting at \( x_0 = 0 \) with \( \alpha = 0.1 \):

• \( x_1 = 0 - 0.1 \times 2(0 - 3) = 0.6 \)
• \( x_2 = 0.6 - 0.1 \times 2(0.6 - 3) = 1.08 \)
• \( x_3 = 1.08 - 0.1 \times 2(1.08 - 3) = 1.464 \)

Each step moves 20% of the remaining distance towards the minimum at \( x = 3 \). Training a neural network is essentially this process repeated billions of times.

Did You Know?

• Leonid Kantorovich shared the 1975 Nobel Prize in Economics for his work on optimal allocation of resources, which grew out of linear programming.
• Soap films naturally solve optimization problems: they form surfaces of minimum area, a fact used by architects such as Frei Otto to design lightweight roofs.
• Honeybees build hexagonal cells; in 1999 Thomas Hales proved that hexagons divide a plane into equal areas with the least total perimeter.

Key Takeaways

• Every optimization problem has decision variables, an objective function and constraints.
• In calculus, optima occur where the derivative is zero or at endpoints; the second derivative tells maximum from minimum.
• Linear programming problems reach their optimum at a corner of the feasible region.
• Lagrange multipliers handle curved constraints, and gradient descent handles problems too large to solve by hand.

Need help setting up or solving an optimization problem for your assignment? Amathify's experts and PhD scholars provide step-by-step solutions via WhatsApp or the Ask a Question form.