NON-LINIER OPTIMIZATION

Presentasi berjudul: "NON-LINIER OPTIMIZATION"— Transcript presentasi:

1 NON-LINIER OPTIMIZATION
CHAPTER 5 NON-LINIER OPTIMIZATION

2 5.1. Introduction Problem estimisasi secara umum MaX (Min) f (X)
s.t X  y F disebut objektif function. X disebut himpunan feasible. Himpunan feasible biasanya digambarkan dengan persamaan atau pertidaksamaan yang disebut kendala. Contoh linier programming problem : MaX ctX s.t. AX = b X ≥ 0 X = {XRn| AX = b, X ≥0} Jika tidak ada kendala maka X = Rn, dan problem ini disebut optimisasi tanpa kendala

3 5.2. Global dan Local Optima
5.2.1 Definisi f = D  R dimana D  Rn. Fungsi f dikatakan memiliki global (absolute) maximum pada X*  D jhj f(X) ≤ f(X*)  X  D. Titik X* disebut global maximizer (or maximizer) dari f dan f(X*) disebut global maximum dari f dan sebaliknya. Global minimum f(X) ≥ f (X*)  X  D. Fungsi f dikatakan memiliki local (relative) maximum pada X*  D jhj terdapat neighborhood N (X*) seperti : f(X) ≤ f(X*)  X  N (X*)  D. Titik X* disebut local maximizer dari f dan f(X*) disebut local maximum dari f. Dan sebaliknya adalah: Fungsi local minimum X*  D jhj f(X) ≥ f(X*)  X  N(X*)  D

4 5.2.2 Contoh Local dan Global Optimum
f(X) X1 X2 X3 X4 X5 X6 D X1, X3, X5 adalah local maximum, X5 global max X2, X4, X6 adalah local minimum, X2 global min

5 5.2.3 Remark (1) Global max digunakan secara relatif to domain D dari f yaitu global maximizer berarti global maximizer to domain (2) Istilah optimum digunakan baik maximum maupun minimum (3) Suatu global maximizer (global minimizer) adalah local maximizer (local minimizer) (4) Local maximum dapat lebih kecil dibanding local minimun (5) Suatu fungsi tidak memiliki baik maximizer ataupun minimizer dalam domainnya. f(X) = X, tidak memiliki max/min dalam interval (0,1) Catatan : Interval ini tidak tertutup meskipun interval tertutup tetapi tidak memiliki batas (bounded) 5.2.4 Theorema Suatu fungsi continuous real-valued didefinisikan atas suatu compact subset S dari Rn memiliki maximizer & minimizer dalam S.

6 5.3 Necessary (or First-Order) Condition untuk Local Optimum
5.3.1 Theorema Andaikan f : D  R, dimana D ≤ Rn, dan andaikan turunan partial dari f  X(S) pada suatu interior point X* dari D X* adalah local optimizer dari f, maka f’(X) = 0. Proof : 5.3.2 Remark (1) Kasus satu variabel dari theorema 5.3.1 Syarat bahwa turunan exist pada local optimizer adalah necessary dalam teorema 5.3.1 contoh : f(X) = X2/3 f’(X) = 2/3X-1/3 f(X) f(X) f’(X*)=0 f’(X*)=0 X* X X* X

7 Memenuhi local minimum pada X*=0 tetapi f’(X*)0.
Fungsi f memiliki local minimum pada X*=0 but f’(0)≠0. Contoh di atas tidak differentiable pada X*=0 (ini berarti bukan sufficient). (3) Syarat suatu X* adalah interior point dari domainnya adalah necessary. Contoh andaikan gambar 5.5 domain D adalah interval tertutup [0,1]. Memenuhi local minimum pada X*=0 tetapi f’(X*)0. Note: X* tidak pada interior point dari D. f(X) f(X) = X2/3 Gambar 5.4 X* X f’(X*)0 Gambar 5.5 D X* 1 X

8 (4) Kebalikan dari teorema 5.3.1 tidak benar Contoh : f(X) = X3
f’(0) = 0 tetapi f tidak memiliki local optimum pada X*=0. Contoh tiga dimensi f(X1,X2) = X12 – X22 Gambar 5.7 gradien f adalah f’(X1,X2) = Pada X*=[0,0]t, f(X*)=0 tetapi untuk titik dekat X* pada sb X1–axis, f(X1,X2)>0 dan titik dekat X* pada sb X2 f(X1,X2)<0. f(X) f’(X)=X3 X Gambar 5.6 2X1 -2X2

9 Consekuensi X*=0 mendekati local maximizer dan local minimizer sehingga titik tersebut disebut saddle point. Gambar 5.7. Saddle point X3 X2 X1 5.3.3 Definisi Titik X* dimana f’(X*)=0 disebut stationary point f. stationary point dan bukan titik optimum disebut saddle point dari f. Dalam pandang theorema titik optimizer f pada domain D ditemukan. (a) Stationary point (figure 5.7.1(a)) (b) The boundary point (figure 5.7.1(b)) (c) f’(X) tidak exist (figure 5.7.1(c))

10 5.4 Sufficient (Second-Order) Condition for
X* X* X* D D D a. f’(X*) = 0 b. f’(X*)  0 c. f’(X*)=n tidak diperoleh Gambar 5.7.1 Kemudian sufficient condition untuk suatu stationary point menjadi local optimizer ketika f setidaknya dapat di diferensialkan 2 kali secara continuous. 5.4 Sufficient (Second-Order) Condition for Local Optima : The One-Variable Case 5.4.1 Theorema Andaikan D ≤ R . f : D  R memiliki continuous second-order derivative atas open interval I yang berisi X*. Andaikan f’(X*)=0. (i) Jika f”(X*) > 0, X* adalah local minimizer dari f (ii) Jika f”(X*) < 0, X* adalah local maximizer dari f

11 Proof : 5.4.2 Contoh Let f : R  R f(X) = 2X3 – 3X2 – 12X + 1 f’(X) = 6X2 – 6X – 12 Setting f’(X) = 0 diperoleh X* = 2, X** = -1 f”(X) = 12 X – 6 f”(2) = 12(2) – 6 = 18 f”(-1) = 12(-1) – 6 = -18 Ini berarti bahwa X* = 2 adalah local maximizer dari f X** = -1 adalah local minimizer dari f 5.4.3 Remark Ketika f”(X*)=0, tidak ada kesimpulan yang dapat dibuat. Kita gunakan theorema berikutnya untuk menggeneralisasikan theorema 5.4.1

12 5.4.4 Theorema Let D C R dan f : D  R memiliki continuous nth-order derivative pada N(X*) C D. Suppose f’(X*) = f”(X*) = … = f(n-1)(X*) = 0 dan f(n)  0 (i) Jika n bilangan genap dan f(n)(X*) > 0, maka X* adalah local minimum dari f (ii) Jika n bilangan genap dan f(n)(X*) < 0 maka X* adalah local maXimum dari f (iii) Jika n bilangan ganjil, maka X* adalah saddle point dari f Proof : f(X) = (X – 5)4 Necessary condition : f’(X) = 4(X-5)3 = 0 Stationary Point : X* = 5 Sufficient condition : f”(X) = 12(X-5)2, f”(5) = 0 f(3)(X) = 24(X–5), f3(5) = 0 f4(X) = 24, F4(5) = 24 5.4.5 C ontoh Kasus ini n=4 dan f4(5) > 0, sehingga X*=5 adalah local minimizer pada f.

13 5.5 Sufficient (or Second-Order) Condition for
Local Optima : The n-Variable Case 5.5.1 Theorema Let D C Rn + f : D  R mempunyai continous second order partial derivative pada N(X*) C D dan f’(X*) = 0. (i) Jika f”(X*) adalah positif definit, X* adalah local minimizer dari f (ii) Jika f”(X*) adalah negatif definit, X* adalah local maximizer dari f (iii) Jika f”(X*) indefinite, X* adalah saddle-point dari f Proof : 5.5.2 Remark Ketika Hessian Matriks f”(X*) adalah positif atau negatif semi definit, tidak ada kesimpulan yang dapat ditarik.

14 5.5.3 Contoh f(X) = -X12 – 3X22 – 2X32 + 2X1 – 12X2 + 8X3 – 5 -2X1 + 2
Gradien dari f : f’(X) = Setting f’(X) = 0, maka titik stationer X* = [ ]t Hessian matriks dari f pada X* adalah f”(X* ) = [f’’ij(X*)] = Memiliki leading principle minor adalah |[-2]| = -2 < 0 = 12 > 0

15 = -48 < 0 Dari theorema bahwa f”(X*) adalah negatif definite. Alternatif, catatan bahwa f”(X*) adalah matriks diagonal, maka eigenvalue adalah elemen diagonal (theorema ). Sejak eigenvalue negatif maka f”(X*) adalah negatif definit (by Theorema ). Sehingga X* adalah local maximizer dari f (by theorema 5.5.1). 5.5.4 Contoh f(X) = 4X13 + X1X2 – 3/2X12 + ½X Gradient dari f adalah 12X12 + X2 – 3X1 X1+ X2 f’(X) = Setting f’(X) = 0 diperoleh titik stationer : 1/3 -1/3 X* = , X** =

16 Hessian matriks dari f adalah
24X f”(X) = Pertimbangankan titik stationer X* maka Hessian matrix dari f pada X* adalah 24(0) f”(X*) = = Matriks Real symmetric dimana eigenvalue adalah 1 = , 2 = 1 – 5 Apabila 1 > 0 dan 2 < 0 sesuai teorema bahwa f”(X*) adalah indefinite. Konsekuensinya, X* adalah saddle-point dari f by theorema 5.5.1 Hessian matrix pada X** adalah 24(1/3) f”(X**) = =

17 Matriks Real symmetric dimana Leading Principal Minors (LPM) adalah
|[5]| = 5 > 0 , = 4 > 0 Theorema f”(X**) adalah positif definite. Alternatif, calculate eigenvalue dari f”(X**); 1 = , 2 = 3 – 5 1>0 dan 2>0, by theorem f”(X**) adalah positif definite. sehingga X** sebagai local minimizer dari f by theorem 5.5.1 5.5.5 Contoh Let f : R2  R didefinisikan dengan f(X) = X12 + X24 Gradient dari f adalah f’(X) = Setting f’(X) = 0 , diperoleh stationary point X* = [ ]t 2X1 4X23

18 5.5.6 Contoh Hessian matriks dari f adalah f”(X) = 2 0
Pada X*, Hessian matriks dari f adalah f”(X*) = Eigenvalue adalah 1=2 dan 2=0 sesuai theorema that f”(X*) adalah positive semidefinite tetapi tidak positif definit sehingga tidak dapat diputuskan. X22 5.5.6 Contoh Let f : R2  R didefinisikan dengan f(X) = X12 – X24 Gradient pada f adalah f’(X) = 2X1 - 4X22

19 Setting f’(X) = 0 , diperoleh titik stasioner
X* = [ ]t Hessian matriks dari f adalah f”(X) = Pada X*, Hessian matriks dari f adalah f”(X*) = Theorem tidak dapat digunakan tetapi f(X*)=0 dan untuk titik X mendekati X* pada X1–axis, f(X)=(X1)2 > 0 sedang untuk titik X mendekati X* pada X2–axis, (f(X)=-(X2)4 < 0. Sehingga X* adalah saddle-point dari f (lihat Danao, ). X22

20 5.5.7 Contoh Profit Maximization
q = output, R(q), C(q), (q). R dan C continuous second-order derivatives. (q) = R(q) – C(q) ’(q*) = R’(q*) – C’(q*) = 0 R’(q*) = C’(q*)  optimal profit R’(q*) < C’(q*)  q   R’(q*) > C’(q*)  q   5.5.8 Contoh The Method least Squares. Garis regresi : Y = 0 + 1X Untuk setiap i i = Yi – (0 + 1Xi) Untuk estimasi 0 dan 1 (i)2 =  (Yi – 0 – 1Xi)2 n i=1 n i=1

21 adalah minimum.  disederhanakan sebagai .
f(0, 1) = (Yi – 0 – 1Xi)2 f’(0, 1) = 0 untuk minimum 2(Yi – 0 – 1Xi)( -1) = 0 2(Yi – 0 – 1Xi)(-Xi) = 0 Maka n 0 + (Xi)1 = Y1 (Xi)0 + (X12)1 = YiXi Normal equation dalam bentuk matrix n Xi Xi Xi2 yi yiXi B0 B1 = Asumsi coefficient matriks adalah non-singular, menggunakan Cramer’s Rule, titik stasioner : nXiYi – (Xi) (Yi) n(Xi)2 – (Xi)2 1* = 1 n 1 n 0* = Yi – (Xi) 1*

22 5.6 Optima of Concave and Convex Functions
to verify * is a global minimizer dari f, dengan melihat Hessian matrix dari f : 2 2Xi 2Xi 2Xi2 2n 2Xi 2Xi 2Xi2 f”(0, 1) = = LPM adalah : 2n 2Xi 2Xi 2(Xi)2 |[2n]| = 2n>0 ; = 4n(Xi)2 – 4(Xi)2 = 4[n(Xi)2 – (Xi)2] = 4[(n-1)(Xi)2 – 2 Xi Xj] >0 Dari persamaan (5-2) Hessian matriks f”(0,1) adalah positive definite and so the stationary point * minimizer f. 5.6 Optima of Concave and Convex Functions 5.6.1 Definisi Let X,Y  Rn, titik Z = X + (1–)Y Dimana 0 ≤  ≤ 1 disebut convex combination dari X dan Y

23 5.6.2 Remark 5.6.3 Definisi 5.6.4 Remarks
Geometrically, the set of all convex combinations of X and Y is the line segment joining X and Y 5.6.3 Definisi A subset C of Rn adalah convex jhj line segment berhubungan dari titik dalam C berada dalam himpunan C. In symbols, C adalah convex jhj X,Y  C, 0 ≤  ≤ (X + (1- )Y)  C 5.6.4 Remarks 1) Himpunan kosong (O) dan himpunan hanya satu elemen adalah convex set 2) Rn adalah convex set y=1/X Convex Set Strictly Convex Not Strictly Convex

24 5.6.5 Theorema 5.6.6 Definisi 3) The following sets are convex :
Not ConveX Set 3) The following sets are convex : (a) The closed half-spaces : H+(P,) = {X  Rn| PtX  } H-(P,) = {X  Rn| PtX ≤ } (b) The hyperplane : H(P,) = {X  Rn | PtX = } (c) The Non-negative orthant : Rn+ = {XRn | X  0} (d) The positive orthant : Rn++ = {XRn|X>0} 5.6.5 Theorema Interseksi dari convex set adalah convex. Proof : 5.6.6 Definisi A subset C of Rn adalah strictly convex jhj X,Y  C, X  Y, 0 <  < [X + (1-)Y]  int (C)

25 5.6.7 Remarks 5.6.8 Definisi 5.6.10 Definisi 5.6.11 Definisi
An open convex set is strictly convex. A closed disk is strictly convex while a closed triangle is not. Intuitively, a closed strictly convex set does not have a flat portion on its boundary. 5.6.8 Definisi A function f : C  R defined on a convex subset C of Rn is said to be concave on C if and only if X,Y  C, 0 ≤  ≤ 1, f(X+(1- )Y) ≥ f(X)+(1- )f(Y) Definisi f : C  R defined on a convex subset C of Rn is said to be strictly concave on C if and only if X,Y  C, X ≠ Y, 0 <  < 1, f(X+(1-)Y) > f(X)+(1- )f(Y) Definisi f : C  R defined on a convex subset C of Rn is said to be strictly convex on C if and only if X,Y  C, X ≠ y, 0 <  < 1, f(X+(1- )Y) < f(X)+(1-)f(Y)

26 Theorema A concave (convex) function is continuous in the interior of its domain Concave function Convex function f((X1)+(1- )X2) f(X2) f(X1) f(X1)+(1- )f(X2) X1 (X1)+(1- )X2 X2 f(X1)+(1-)f(X2) f(X1) f(X2) f(X1+(1- )X2 X1 X1+(1-)X2 X2

27 Neither concave nor convex functions
Concave function Convex function Neither concave nor convex functions Figure 5.11 Y=ln(X) Y X Strictly concave function Y Y=1/X X Strictly convex function

28 5.6.13 Theorema 5.6.14 Definisi 5.6.15 Remark
f : C  R dimana C adalah convex subset of Rn if f is concave on C then UCf() = {X  C|f(X) ≥ } is convex for every   R if f is convex on C then LCf() = {X  C|f(X) ≤ } Definisi f : D  R adalah function defined on a subset D of Rn. - UCf() = {X  D| f(X) ≥ ,   R} disebut upper contour set of f - LCf() = {X  D| f(X) ≤ ,   R} disebut lower contour set of f - Cf() = {X  D| f(X) = ,   R} disebut contour (or level) set of f Remark

29 Theorem f : C  R adalah continuously differentiable on convex subset of Rn (i) f adalah concave on C if and only if X,Y  C, f(Y) - f(X) ≤ [f’(X)]t (Y-X) (ii) f adalah convex on C if only if X,Y  C, f(Y) - f(X) ≥ [f’(X)]t (Y-X) Theorem f : I  R adalah continuously differentiable on open interval I f adalah concave on I if only if X,Y  I, f(Y) - f(X) ≤ f’(X) (Y-X) (ii) f adalah convex on I if only if X,Y  I, f(Y) - f(X) ≥ f’(X) (Y-X)

30 Theorem f : C  R adalah continuously differentiable on open convex subset C of Rn f adalah strictly concave on C if only if X,Y  C, X ≠ Y f(Y) - f(X) < [f’(X)]t (Y-X) (ii) f adalah strictly convex on C if only if X,Y  C, X ≠ Y f(Y) - f(X) > [f’(X)]t (Y-X) Theorem f : C  R adalah twice continuously differentiable on open concex subset C of Rn then f is concave on C jhj Hessian matrix f”(X) is negatif semidefinite on C f is convex on C jhj Hessian matriX f”(X) is positive

31 Theorem f : I  R be twice continuously differentiable on an open interval I X{Ci | if the hessian matriks f”(X) adalah negatif (i) f is concave on I jhj f”(X) ≤ 0 on I (ii) f is conveX on I jhj f”(X) ≥ 0 on I Theorem f have continuous second-order partial derivative on an open convex subset C of Rn If Hessian matriks f”(X) is negative definite for every X  C, then f is strictly concave on C (ii) If Hessian matriks f”(X) is positive definite for every X  C, then f is strictly convex on C Proof : Polak

32 5.6.23 Remark 5.6.24 Theorema f : R  R defined f (X) = X4.
f is continuous second-order derivative on R. f is strictly convex on R. But f”(0) = 0 which is not positive definite. Theorema f : I  R adalah twice continuously differentiable on an open interval I (i) If f”(X) < 0 for every X  I then f is strictly concave on I (ii) If f”(X) > 0 for every X  I then f is strictly convex on I Sum and compositions of concave and convex function

33 Theorem f : C  R dan g : C  R where C is a convex subset of Rn (i) If f dan g adalah concave on C then (a) f + g is concave on C (b) f is concave on C for each  > 0 (c) f is convex on C for each  < 0 (ii) If f dan g adalah convex on C then (a) f + g is convex on C (b) f is convex on C for each  > 0 (c) f is concave on C for each  < 0 Theorem f : C  R dan g : C  R; dimana C is a convex subset on Rn (i) If f dan g adalah strictly concave on C then (a) f + g is strictly concave on C (b) f is strictly concave for each  > 0 (c) f is strictly convex for each  < 0 (ii) If f dan g adalah strictly convex on C then (a) f + g is strictly convex on C (b) f is strictly convex on C for each  > 0 (c) f is strictly concave on C for each  < 0

34 Theorem Let : f : C  R be defined on a convex subset C of Rn such that f(C) is convex. Let g : f(C)  R be defined on f(C) (i) If f is concave on C and g is concave and increasing of f (C), then the composition of f and g is concave on C. (ii) If f is convex on C and g is convex and increasing on f (C), then the composition of f and g is convex on C. Theorem Let : f : C  R be defined on a convex subset C of Rn such that f(C) is convex. Let g : f(C)  R be defined on f(C) (i) If f is strictly concave on C and g is strictly concave and increasing of f (C), then the composition of f and g is strictly concave on C. (ii) If f is strictly convex on f(C), then the composition of f and g is strictly convex on C.

35 5.6.29 Example (1) f(X) = X12 + X1X3 – X2 + X22 + X2X3 + 3X32
Hessian matrix of f is f”(X) = LPM are [2] > 0 2 0 0 2 = 4 > 0 = 20 > 0 f”(X) is positive definite for each X  R3. Therefore, f is strictly conveX on R3 .

36 (2) Consider the production function f : R++2  R
f(L,K) = L – K, 0<,<1 , +<1 The gradient of f is L-1 K L K -1 f’(L,K) = The Hessian matrix of f is (-1)L-2K L-1K -1 L-1K (-1)LK -2 f”(L,K) = and its Leading Principal Minors are |[ (-1)L-2K]| = (-1)L-2K < 0 (-1)L-2K  L-1K  -1 L-1K -1 (-1)LK -2 = [(-1)(-1) – 22]L2-2K2  -2 =  [1-(+)]L2-2K2  -2 > 0 If follows that this function is strictly concave on the positive quadrant.

37 (3) Let f : R++  R be defined by
f(X) = Ln(X) f”(X) = < 0 For all X  R++. It follows from theorem that f is strictly concave on R++ -1 X2 (4) Let f : R+  R defined by f(X) = X f’(X) = X-1 f”(X) = (-1)X-2 Hence, f is strictly concave if  < 1 f is strictly convex if  > 1 f is concave and convex if  = 1 Hence, f is concave if  ≤ 1 f is convex if  ≥ 1

38 (5) Let f : R++n  R defined by
f(X) = CtX = CiXi then the function g : R++n  R definey by g(X) = ln (CtX) = ln ( CiXi) is concave on R++n, since f is concave and ln is a concave and incrasing function. n i=1 n i=1 (6) Let f : R++2  R defined by f(X) = 1 ln (X1) + 2 ln (X2) , 1,2 > 0 claim that f is strictly concave on R++2

39 5.6.32 Example : Consider example 5.6.29(1)
Theorem Every local maximizer of a concave function is a global maximizer Every local minimizer of a convex function is a global minimizer Proof : Theorem A local maximizer of strictly concave function is unique A local minimizer of strictly convex function is unique Example : Consider example (1) f(X) = X12 + X1X3 – X2 + X22 + X2X3 + 3X32 It was shown that f is strictly convex on R3. The stationary point are obtained by setting f’(X) = 0, i.e. 2X X3 = 0 2X2 + X3 = 1 2X1 + X2 + 6X3 = 0 Stationary point : X* = [1/20, 11/20, 2/20]t

40 Hessian matriks f’’(X. ) is positive definite then theorem 5. 5. 1, X
Hessian matriks f’’(X*) is positive definite then theorem 5.5.1, X* is a local minimizer of f. But f is strictly convex: hence, X* is the unique global minimizer of f by theorem and theorem Example The Linier Programming Problem min Co + CtX S.t AX ≤ b , A is m x n , X  Rn f is strictly conveX on R3. The stationary point f’(X) = 0 The objective function of the Linier Programming (LP) is both concave and convex. The feasible region X = {X  Rn|AX ≤ b, X ≥ 0} is convex since it is the intersection of convex set X = H  R+n Dimana H = {X  Rn|AX ≤ b} and R+n = {X  Rn|X ≥ 0} Consequently, any optimal solution of the LP is a global optimal solution. Obviously, This is also true of the maximization problem.

41 5.7 The Optima of Quasiconcave and Quasiconvex Function
5.7.1 Definition Function f : C  R defined on convex subset C of Rn is said to be quasiconcave on C if and only if 5.7.2 Definition Function f : C  R defined on convex subset C of Rn is said to be quasiconvex on C if and only if 5.7.3 Definition Function f : C  R defined on convex subset C of Rn is said to be strictly quasiconcave on C if and only if 5.7.4 Definition Function f : C  R defined on convex subset C of Rn is said to be strictly quasiconvex on C if and only if 5.7.8 Corollary Every concave function is quasiconcave Every convex function is quasiconvex

42 5.7.9 Corollary 5.7.10 Theorem 5.7.11 Theorem 5.7.12 Example
A linear function is both quasiconcave and quasiconvex. Theorem (i) A strictly concave function is strictly quasiconcave (ii) A strictly convex function is strictly quasiconvex. Theorem Let : f : C  R be continuous on a strictly convex subset C of Rn. If is strictly quasiconcave on C, then the upper contour set UCf () is strictly convex for every   R. Example The Normal Distribution Function Theorem A differentiable function f : C  R defined on open convex set C C Rn is quasiconcave on C if and only if X,Y  C, f(X) ≥ f(Y) [f’(Y)]t (X-Y) ≥ 0 (ii) A differentiable function f : C  R defined on open convex set C C Rn is quasiconvex on C if and only if X,Y  C, f(X) ≤ f(Y) [f’(Y)]t (X-Y) ≤ 0

43 5.8 Constrained Optimization
The general form of the constrained optimization problem may be expressed as follows : Max (Min) f (X) S.t Gk (X) ≥ 0 , k = 1, 2, 3, …, m Where X  Rn. The function f is objective function and the in-equalities called the constraints. A vector X satisfies the constraints called a feasible solution and the set of feasible solutions is called feasible set or feasible region. A feasible solution that maximizes (minimizes) the value of the objective function on the feasible region is called maximizer (minimizer). The term optimizer or optimal solution refers to either maximizer or minimizer. The distinction between a constrained optimization problem and an un-constrained problem can be seen fram the geometry of two-variable problem with a single equality constraint. Suppose that the problem is : Max f(X1,X2) S.t. P1 X1 + P2 X2 = Y

44 5.9 Optimization With One Equality Constraint : The Two-Variable Case
Solution by Direct Substitution Consider the problem Max (Min) f (X1, X2) S.t g (X1, X2) = 0 Dimana f dan g adalah diferensiabel. If, from the constraint g(X1, X2) = 0, it is possible to express one variable, say X2 in terms of the other variable then write X2 = h(X1) and substitute this in f (X1, X2). The problem reduces to the un-constrained problem of optimizing f [ X1, h(X1)]. 5.9.1 Example Min (X1 – 1)2 + (X2)2 S.t X1 + X2) = 4 Let f (X1, X2) = (X1 – 1)2 + (X2)2 From the constraint, we get X2 = h(X1) = 4 - 2(X1) X1* = 9/5 X2* = 4 – 2(9/5) = 2/5

45 5.9.2 Remark The solution by direct substitution is particularly useful if the constraint is linear since it is easy to express one variable in terms of the other variables. When the constraint is non-linear, it may be difficult or impossible to obtain such an explicit function. This limits the usefulness of this method. On the other hand, the graphical method is limited to two-variable problems. These limitations are not possessed by a third method of solution called the Lagrange Multiplier Method. 5.9.3 Definition Max (Min) f (X) S.t g (X1) = b, X  R2 The function L defined by L(,X) = f (X) + [g(X) – b] Is called the Lagrangean of the problem and the scalar  is called the Lagrange multiplier

46 5.9.4 Remark The Lagrangean is also written as
L(,X) = f (X) – [g(X) – b] (2) The Gradient of L is L’ (,X) g (X) – b L1’ (,X) f1’ (X)+g1’(X) L2’ (,X) f2’ (X)+g2’(X) L’ (,X) = = and the Hessian matrix of L is L1’’ (,X) L11’’ (,X) L21’’ (,X) L’’ (,X) L1’’ (,X) L2’’ (,X) L2’’ (,X) L12’’ (,X) L22’’ (,X) L’’ (,X) = g1’(X) g2’(X) g1’(X) L11’’(,X) L21’’(,X) g2’(X) L12’’ (,X) L22’’ (,X) =

47 5.9.5 Theorem (Necessary or First-Order Condition) Given the problem Max (Min) f (X) S.t g (X) = b, X  R2 Where f and g have continuous partial derivatives. Let X* be an optimizer of f on the feasible set and suppose that g’(X*)  0, j=1,2. Then there exists a scalar * such that L’(*, X*) = 0 Proof : 5.9.6 Theorem (Sufficient or Second-Order Condition) Given the problem Max (Min) f (X) s.t. g(X) = b, X  R2 Where f and g have continuous second-order partial derivatives. Let X* and * satisfy L’(*,X*) = 0 and suppose that gj’(X*)  0, j = 1, 2

48 (i) If |L”(*,X*)| < 0 , Then X* is a local minimizer
(ii) If |L”(*,X*)| > 0 , Then X* is a local maximizer Proof : For notational convenience, we will suppress the arguments of each function; e.g., g1(X) will be written simply is g1’. From the constraint, the total differential is g1’dX1+ g1’ dX2 = 0, from which we get dX2 dX1 g1’ g2’ = – Let Y = f(X1,X2) Then dY = f1’dX1+ f2’dX2 Hence, = f1’ + f2’ dY dX1 dX2 dX1 Subtituting …. We know that if < 0 at X*, then X* is a local maximizer. Hence, if the determinant |L’’(*,X*)| > 0 , then X* is a local maximizer. d2Y dX12

49 5.9.7 Example Min (X1 – 1)2 + (X2)2 s.t. 2X1 + X2 = 4
Lagrangean: L(,X) = (X1– 1)2 + (X2)2 + (2X1 + X2 – 4) F.O.C. L’(,X) = 2X1 + X2 – 4 = 0 L1’(,X) = 2(X1– 1) + 2 = 0 L2’(,X) = 2X2 +  = 0 X1* = 9/5 , X2* = 2/5 , * = – 4/5 S.O.C |L”(u*,X*)| = = -10 < 0 By theorem X* is local minimizer on feasible set. Hessian matrix of the objective function is f”(X) = Which is positive definite for every XR2  f is convex. The objective function is strictly convex. X* is the unique global minimizer. 2 0 0 2

50 5.9.8 Example Max f(X1, X2) = X1X2 s.t. P1X1 + P2X2 = Y , P1,P2,Y > 0 Lagrangean: L(u,X) = X1X2 + (P1X1 + P2X2 – Y) F.O.C. L’(,X) = P1X1 + P2X2 – Y = 0 L1’(,X) = X2 + P1 = 0 L2’(,X) = X1 + P2 = 0 X1* = Y/2P1 , X2* = Y/2P2 , * = – Y/2P1P2 0 P1 P2 S.O.C |L”(u*,X*)| = P = 2P1P2 > 0 P By theorem X* is a local maximizer on the feasible set. X* the unique global maximizer on the feasible set. They can not opposite signs since their objective function value would be negative which can not be optimal since VOF (X*) = X1*X2* = > 0 X* is the unique global maximizer on the feasible set. y2 4P1P2

51 5.10 Optimization With One Equality Constraint : The n-Variable Case
Definition Given the problem Max(min) f(X) s.t g(X) = b , X  Rn Lagrangean : L(,X) = f(X) +  [(g(X) – b] Remark 1) The gradient of L is L’(,X) L1’(,X) . Ln’(,X) g(X) – b f1’(X) + g’1(X) . fn’(X) + gn’(X) L’(,X) = =

52 5.10.3 Notation 2) Hessian matriz of L is 0 g1’(X) g2’(X) … gn’(X)
g1’(X) L11”(,X) L12”(,X) … L1n”(,X) g2’(X) L21”(,X) L22”(,X) … L2n”(,X) gn’(X) Ln1”(,X) Ln2”(,X) … Lnn”(,X) L’’(,X) = Notation Leading principal submatrices of order k of the Hessian matrix of L will be denoted by Lk”(,X) L1’’ (,X) = 0 g1’(X) g1’(X) L 11”(,X) L2” (,X) =

53 5.10.4 Theorem 0 g1’(X) g2’(X) g1’(X) L11”(,X) L12 ”(,X) L3”(,X) =
etc L3”(,X) = Theorem (Necessary of First-Order Condition). given the problem Max(min) f(X) s.t g(X) = b , X  Rn Dimana f dan g have continuous FOC partial derivatives. Let X* be an optimizer over the feasible set such that g’j(X*)  0, j= 1,2, …,n. Then there exists a scalar * such that L’ (*,X*) = 0

54 Theorem (Sufficient or Second-Order Condition). Given the problem Max (min) f(X) s.t. g(X) = b , X  Rn Where f and g have continuous second-order partial derivatives. Suppose that X* and * satisfy FOC, i.e. L’(*,X*) = 0. If |L3”(*,X*)| < 0 , |L4”(*,X*)| < 0, …, |Ln”(*,X*)| < 0 , then X* is a local minimizer of f on the feasible set. (ii) If |L3”(*,X*)| > 0 , |L4”(*,X*)| < 0 , |L”5(*,X*)| > 0, …, (-1)n|Ln”(*,X*)| > 0 , then X* is a local maximizer of f on the feasible set. Proof : Gue and Thomas (1968)

55 5.10.6 Example Max (min) X12 + X1X2 + 2X22 + X32
s.t. X1 – 3X2 – 4X3 = 16 Lagrangean : L(,X) = X12 + X1X2 + 2X22 + X32 +  (X1 – 3X2 – 4X3 – 16) First Order Condition : X1 – 3X2 – 4X3 – 16 2X1 + X2 +  X1 + 4X2 + 3 2X3 – 4 L’ (,X) = = This system of equations yields the solution X1* = 4, X2* = – 4, X3* = – 8, * = – 4

56 Hessian Matrix of L : L”(,X) = S.O.C. : |L3”(*,X*)| = = -28 < 0 |L4”(*,X*)| = = -168 < 0 Hence, X* is a local minimizer on the feasible set. The Hessian Matrix of f is : f”(X) =

57 5.11 Optimization With Several Equality
whose eigenvalues are : 1 = 2 , 2 = , 3 = 3 – 2 Which are all positive. Hence, f”(X) is positive definite for all X R3. That f is strictly convex on R3. This implies X* is the unique global minimizer of f on the feasible set. 5.11 Optimization With Several Equality Constraints : The n-Variable Case Definition Given The Problem Max (min) f(X) s.t gk(X) = bk k = 1,2 …,m; m < n , X  Rn The function L defined by L(,X) = f(X) +  k (gk(X) – bk) is called the Lagrangean of the problem. The variables 1, 2, …, n are called the lagrange multipliers. n k=1

58 Remark The gradient of L, with the arguments of the functions suppressed, is : g1 – b1 g2 – b2 . gn – bm f X1 gk X1 L’ = + k . f Xn gk Xn + k

59 The Hessian Matrix of L is :
0 … 0 g1 Xi g1 Xn … . . gm X1 … gm Xn L’ = g1 X1 … gm X1 L11” … L1n” Ln1” … Lnn” . . … g1 Xn gm Xn Note that the matrix L’’ has dimension (m+n) x (m+n). The Leading Principal Sub Matrix (LPSM) of order k will be denoted by Lk”.

60 5.11.3 Theorem  0 (Necessary or F.O.C.). Given the problem
Max (min) f(X) s.t. gk(X) = bk , k = 1,2, …, m m < n , X  Rn Where f and gk (k = 1,2, …,m) have continuous first-Order Partial Derivatives. Let X* be an optimizer of f over the feasible set and suppose that the Jacobian determinant g1(X*)  X1 … g1(X*) Xn . .  0 gm(X*) X1 gm(X*) Xn … Then there exist scalars k* (k=1,2,…,m) such that L’(*, X*) = 0 Proof : Panik (1976).

61 5.11.4 Theorem (Sufficient or S.O.C.). Given the problem
MaX (min) f (X) s.t gk(X) = bk , k = 1,2, …,m , m < n X  Rn Where f and gk (k = 1,2,…,m) have continuous Second-Order Partial Derivatives. Suppose X* and  satisfy the F.O.C., i.e. L’ (*, X*) = 0 (i) If |L2m+1”(*,X*)| , |L2m+2”(*,X*)| ,…, |Lm+n”(*,X*)| have the same sign as (-1)m, then X* is a local minimizer of f on the feasible set. (ii) If |L2m+1”(*,X*)| , |L2m+2”(*,X*)| ,…, |Lm+n”(*,X*)| Alternate in sign with the sign of |L2m+1(*,X*)| being that of (-1)m+1 , then X* is a local maximizer of f on the feasible set. Proof :