Non Linear Programming

Nonlinear Programming

Contents  

Terminologies Types of function  Concave  Convex



Types of nonlinear programming problems  Unconstrained  Constrained

 

Quadratic programming with WinQSB Nonlinear programming with WinQSB

Terminologies    

Feasible region Maxima 

Local



Global

Minima 

Local



Global

Critical Point

Feasible Region Set of all solutions that satisfy all of the constraints.

Maxima It is a point such that the value of the function is greater than all other function values. f(d) > f(x) d is maximum point

Minima It is a point such that the value of the function is less than all other function values. f(c) < f(x) c is minimum point

Critical Point Those points at which maxima or minima can occur. f(x) Global max. f(d)
f(d)>f(x) Local max.

Global min.

Optimization 

n decision variables



feasible region



minimize or maximize objective function

Definition A general optimization problem is to select n decision variables x1, x2, . . . , xn from a given feasible region in such a way as to optimize (minimize or maximize) a given objective function f (x1, x2, . . . , xn) of the decision variables.

NonLinear Programming 



Objective Function (Always Nonlinear) Constraints (May Be Nonlinear / Linear)

Example Max./Min. 2X12+3X22 S.T. X1+X2<3 2X12+X22>5

NLP Problem The problem is called a nonlinear programming problem (NLP)

if

the objective function is nonlinear and/or the feasible region is determined by nonlinear/linear constraints.

Mathematically Maximize f (x1, x2, . . . , xn) non-linear subject to: g1(x1, x2, . . . , xn) ≤ b1 L/non-L ... ... gm(x1, x2, . . . , xn) ≤ bm

Applications  Data networks – Routing  Production planning  Resource allocation  Computer-aided design  Solution of equilibrium models  Data analysis and least squares formulations  Modeling human or organizational behavior

Functions

Functions Types related to nonlinear programming:

Convex  Concave 

Convex Function 

Straight Line



Above or on the function

Definition When a straight line is drawn between any two points on the graph of the function, the line lies on or above the function.

Graphically

Concave Function 

Straight Line



Below or on the function

Definition When a straight line is drawn between any two points on the graph of the function, the line lies below or on the function.

Graphically

Convex Set 

All the points are in the set

A set S is convex if any point on the line segment connecting any two points in the set is also in S.

Graphically The figure shows examples of convex sets in two dimensions.

Non-Convex Set

If a set does not satisfy the requirements of convex set, it is a nonconvex set.

Graphically The figure shows examples of nonconvex sets in two dimensions.

Not e

An important issue in nonlinear programming is whether the feasible region is convex or not…………..?

Feasible Region Issues  When all the constraints of a problem are linear or convex, the feasible region is a convex set.  If the objective function is convex and the feasible region defines a convex set, every local minimum is a global minimum.

Feasible Region Issues If the feasible region of a problem is a nonconvex set, a local minimum may or may not be global minimum 

Test For Concavity SSuppose that ‘f’ is twice differentiable on the open interval I. If f”(x)>0. then ‘f’ is concave upward at each point of I. If f”(x)<0. then ‘f’ is concave downward at each point of I.

Point of Inflection  A point at which f ”(x)=0 may or may not be a point where the function changes from concave upward on one side to concave downward to the other side. But, if the concavity does change in this manner then, such point is called ‘point of inflection’’

Properties of Functions f(x) is convex, if f”(x) 0 f(x) is concave, if f”(x) 0 f(x) is convex, if Hessian has all principal minors non-negative f(x) is concave, if Hessian has principal minors with sign (-1)k, k=1,2,………

Properties of Functions For Maximization Problem, the feasible region is convex & the function is concave. For minimization problem: The feasible region is convex & the function is also convex.

Unconstrained Problems

Types of NLP Problems

 Unconstrained problems  Constrained problems

Unconstrained Problems 

Unconstrained extreme points  Newton Raphson



Gradient Method  Steepest Ascent Descent

GENERIC UNCONSTRAINTS PROBLEM Minimize/Maximize

f(x);

 no constraint on the decision variable x.  decision variables x are continuous.  Non-negativities does not count towards constraint.

Example A retailer is planning her yearly inventory strategy for a commodity which sells a steady rate thorough the year and for which she estimates a yearly demand of D units. The storage cost is $S per unit and the cost of ordering is $C per order. Assuming that there is no lead time,how many orders and of which size must be placed to reduce the total inventory costs?

Model Suppose the retailer places an order of size x, Then must place D/x

orders at a cost C*D/x

The average inventory is x/2 therefore the holding costs are S/2 * x Optimization problem Minimize S/2* x + C*D/x Non-negativity x ≥ 0

Unconstrained Extreme Points Necessary Condition Xo is extreme point f(X0) = 0

if

Unconstrained Extreme Points Sufficient Condition 

Xo is extreme point Hessian matrix

Positive definite when X0 is a minimum point

Negative definite when X0 is a maximum point

Unconstrained Extreme Points TThe necessary condition for X0 to be an extreme point of f(X) is that f(X0) = 0 SThe Sufficient condition for a stationary point X0 to be extreme point is that the Hessian matrix evaluated at X0 is (a) Positive definite when X0 is a minimum point (b) Negative definite when X0 is a maximum point.

Example Consider the function f(x1, x2, x3) = x1 + x1x2 + 2x2 + 3x3 – x12 – 2x22 – x32 The necessary condition f(X0) = 0 gives f/x1 = 1 + x2 –2x1 = 0 f/x2 = x1 + 2 –4x2 = 0 f/x3 = 3 – 2x3 = 0 The solution is given by x0 = {6/7, 5/7, 3/2}.

...Contd. For sufficiency, we evaluate H=

2f/x12 2f/x1x2 2f/x1x3 2f/x2x1 2f/x22 2f/x2x3 2f/x3x1 2f/x3x2 2f/x32 = -2 -1 0

-1 -4 0

0 0 -3

The principal minor determinants of H have values –2, 7 and –21 respectively indicating that {6/7,5/7, 3/2} represents a maximum point.

Newton-Raphson Method f(x) = 0 is difficult to solve Iterative procedure  Initial Point Xo  Next point Xk+1 Xk+1 = Xk – f(Xk) / f ’(Xk)

Newton-Raphson Method The necessary condition, f(x) = 0, may be difficult to solve numerically. The Net-Raphson method is an iterative procedure for solving simultaneous nonlinear equations The idea of the method is to start from an initial point X 0. By using the foregoing equations, a new point X k+1 is determined from Xk. the procedure end with Xm as the solution when Xm  Xm+1 The relationship between Xk and Xk+1 is Xk+1 = Xk – f(Xk) / f ’(Xk) Where k =0,1,2,………….

Steepest (Ascent/Descent) Basic paradigm of steepest methods is as follows • • • •

Start with an initial point x0 Choose a direction d0 = f’(x) Choose a step size λ0 Update the solution x1 = x0 + λ0 d0

If stopping criterion is met stop; Else repeat steps 2-4 with the new point x1

Constrained Problems

Constrained Problems  What are Constrained Problems  KKT Conditions  Separable programming  Quadratic programming

Constrained Problems Constraints on the decision Variables. Constraints gi(x) can be both linear and nonlinear

Mathematically Minimize/Maximize f(x); Subject to gi(x)=bi; gi(x)≤bi; gi(x)≥bi; Where i = 1,……, m

Real World Problems

Water Resource Planning In regional water planning, sources emitting pollutants might be required to remove waste from the water system. Let x j be the pounds of Biological Oxygen Demand (an often-used measure of pollution) to be removed at source j. One model might be to minimize total costs to the region to meet specified pollution standards

Mathematical Form Minimize n

Σ fj (xj ), j=1

Subject to: n

Σ aij xj ≤ bi (i = 1, 2, . . . ,m) j=1

( j = 1, 2, . . . , n) and 0 ≤ xj ≤uj

. . . where  fj (xj ) = Cost of removing xj pounds of Biological Oxygen Demand at source j ,  bi = Minimum desired improvement in water quality at point i in the system,  aij = Quality response, at point i in the water system, caused by removing one pound of Biological Oxygen Demand at source j ,  uj = Maximum pounds of Biological Oxygen Demand

Inequality Constrained Problems

The necessary conditions for inequality constrained problem were first published in W. Krush's master's thesis, although they became renowned after a seminal conference paper by Harold W. Kuhn and Albert W. Tucker

KRUSH-KUHN-TUCKER Conditions

Known as KKT Conditions 

Set of equations/inequalities solution must satisfy

KRUSH-KUHN-TUCKER Conditions A system of equations and inequalities which the solution of a NLP problem must satisfy when the objective function and the constraint functions are differentiable It is a generalization of method of Lagrange multipliers.

Example where f(x) is the function to be minimized,

are the inequality constraints

are the equality constraints, and m and l are the number of inequality and equality constraints, respectively

KKT Conditions  0 f(X) - g(X) + = 0 i gi(X) = 0 g(X)  0 (The same conditions apply to minimization as well, with the additional restriction that  must be non-positive.)

Separable Function A function f(x) is separable if it can be expresses as the sum n

f(x) =Σ fi(xi) i=1

of single variable functions fi(xi).

Mathematically A constrained nonlinear problem of the form Min/Max f(x) Subject to gi(x)=bi; gi(x)≤bi; gi(x)≥bi; Where i = 1,……, m is separable problem if every gi is separable.

Example Max f (x)= 20x1+16x2 − 2x12− x22− (x1 + x2)2 subject to: x1 + x2≤ 5, x1 ≥0, x2 ≥ 0. As stated, the problem is not separable, because of the term (x1 + x2)2 in the objective function. Letting x3 = x1 + x2, though, we can re-express it in separable form as Contd...

Max f (x) = 20x1+16x2 − 2x12− x22−x32 The objective function is now written as f (x) = f1(x1) + f2(x2) + f3(x3), subject to x1 + x2≤ 5, x1 + x2 − x3 = 0, x1 ≥0, x2 ≥ 0, x3 ≥0. where f1(x1) = 20x1 − 2x12 f2(x2) = 16x2 − x22 And f3(x3) = −x32

Quadratic Programming



Objective Function – Maximization



Constraints - linear

Quadratic Programming Quadratic programming concerns the maximization of a quadratic objective function subject to linear constraints,

Example Max f (x) = 20x1+16x2 − 2x12− x22− (x1 + x2)2 subject to: x1 + x2≤ 5 x1 ≥0, x2 ≥ 0 Expanding (x1 + x2)2 as x12 + 2x1 x2 + x22 and incorporating the factor 1/2 , we rewrite the objective function as: Contd...

Max f (x) = 20x1+16x2+1/2(−6x1x1−4x2x2−2x1x2−2x2x1)

so that q11 q12 q21 q22

= = = =

−6 −2 −2 −4

Quadratic Programming using WinQSB

Step-1

Quadratic Programming

WinQSB

Step-2

Problem Specifications

Step-3

Step-4

Step-5

RUN

Step-6

Step-7

Step-8

Graph

Step-9

Step-10

Step-11

Step-12

Step-13

Step-14

Step-15

Step-16

Step-17

Step-18

Step-19

Step-20

Non-Linear Programming Using WinQSB

New problem

Solve the problem

minimize

Thank You!