Image Registration & Tracking dengan Metode Lucas & Kanade

Presentasi berjudul: "Image Registration & Tracking dengan Metode Lucas & Kanade"— Transcript presentasi:

1 Image Registration & Tracking dengan Metode Lucas & Kanade
Sumber: Forsyth & Ponce Chap. 19, 20 Tomashi & Kanade: Good Feature to Track

2 Feature Lucas-Kanade(LK)
Extraksi feature dengan metode LK ini adalah sangat populer dalam aplikasi computer vision. Feature diekstraksi dengan mengambil informasi gradient image. Selanjutnya feature ini bisa dimanfaatkan untuk Image registration, yg. Selanjutnya diugnakan utk. tracking, recognition, dan lain-lain Pemilihan feature image yang tepat adalah sangat menentukan keberhasilan proses recognition, tracking, etc.

3 Sejarah Perkembangan LK
Lucas & Kanade (IUW 1981) BAHH ST S BJ HB BL G SI CET SC Bergen, Anandan, Hanna, Hingorani (ECCV 1992) Shi & Tomasi (CVPR 1994) Szeliski & Coughlan (CVPR 1994) Szeliski (WACV 1994) Black & Jepson (ECCV 1996) Hager & Belhumeur (CVPR 1996) Bainbridge-Smith & Lane (IVC 1997) Gleicher (CVPR 1997) Sclaroff & Isidoro (ICCV 1998) Cootes, Edwards, & Taylor (ECCV 1998) LK

4 Image Registration

5 Penerapan metode LK

6 Penerapan pada aplikasi:
Stereo LK BAHH ST S BJ HB BL G SI CET SC

7 Penerapan pada aplikasi:
Stereo Dense optic flow LK BAHH ST SC S BJ HB BL G SI CET

8 Penerapan pada aplikasi:
Stereo Dense optic flow Image mosaics LK BAHH ST SC S BJ HB BL G SI CET

9 Penerapan pada aplikasi:
Stereo Dense optic flow Image mosaics Tracking LK BAHH ST SC S BJ HB BL G SI CET

10 Penerapan pada aplikasi:
Stereo Dense optic flow Image mosaics Tracking Recognition ? LK BAHH ST SC S BJ HB BL G SI CET

11 Derivasi Rumusan Lucas & Kanade #1

12 rumusan L&K 1 I0(x)

13 rumusan L&K 1 I0(x+h) h I0(x)

14 rumusan L&K 1 h I0(x) I(x)

15 rumusan L&K 1 h I0(x) I(x)

16 rumusan L&K 1 I0(x) I(x) R

17 rumusan L&K 1 I0(x) I(x)

18 rumusan L&K 1 h0 I0(x) I(x)

19 rumusan L&K 1 I0(x+h0) I(x)

20 rumusan L&K 1 I0(x+h1) I(x)

21 rumusan L&K 1 I0(x+hk) I(x)

22 rumusan L&K 1 I0(x+hf) I(x)

23 Derivasi Rumusan Lucas & Kanade #2

24 E(h) S [ I(x) - I0(x) - hI0’(x) ]2
rumusan L&K 2 Sum-of-squared-difference (SSD) error E(h) = S [ I(x) - I0(x+h) ]2 x e R E(h) S [ I(x) - I0(x) - hI0’(x) ]2 x e R

25 S I0’(x)2 S 2[I0’(x)(I(x) - I0(x) ) - hI0’(x)2] = 0
rumusan L&K 2 S 2[I0’(x)(I(x) - I0(x) ) - hI0’(x)2] x e R = 0 S I0’(x)(I(x) - I0(x)) x e R h S I0’(x)2

26 S S w(x) S I0’(x)2 w(x)[I(x) - I0(x)] I0’(x) h S I0’(x)[I(x) - I0(x)]
Perbandingan h w(x)[I(x) - I0(x)] S w(x) x S I0’(x) S I0’(x)[I(x) - I0(x)] h S I0’(x)2 x

27 S S w(x) S I0’(x)2 w(x)[I(x) - I0(x)] I0’(x) h S I0’(x)[I(x) - I0(x)]
Perbandingan w(x)[I(x) - I0(x)] S I0’(x) x h S w(x) x S I0’(x)[I(x) - I0(x)] x h S I0’(x)2 x

28 Generalisasi metode Lucas-Kanade

29 Rumus Original S [ ] E ( h ) = I ( x + h ) - I0 ( x ) 2 x e R

30 S [ ] E ( h ) = I ( x + h ) - I0 ( x ) 2 Rumus Original
Dimension of image S [ ] E ( h ) = I ( x + h ) - I0 ( x ) 2 x e R 1-dimensional LK BAHH ST S BJ HB BL G SI CET SC

31 S [ ] E ( h ) = I ( x + h ) - I0 ( x ) 2 Generalisasi 1a
Dimension of image S [ ] E ( h ) = I ( x + h ) - I0 ( x ) 2 x e R 2D: LK BAHH ST S BJ HB BL G SI CET SC

32 S [ ] E ( h ) = I ( x + h ) - I0 ( x ) 2 Generalisasi 1b
Dimension of image S [ ] E ( h ) = I ( x + h ) - I0 ( x ) 2 x e R Homogeneous 2D: LK BAHH ST SC S BJ HB BL G SI CET

33 Apakah iterasi bisa konvergen?
Permasalahan A Apakah iterasi bisa konvergen? LK BAHH ST S BJ HB BL G SI CET SC

34 Permasalahan A Local minima:

35 Permasalahan A Local minima:

36 S I0’(x)2 -S I0’(x)(I(x) - I0(x)) h Permasalahan B Zero gradient:
x e R h is undefined if S I0’(x)2 is zero x e R S I0’(x)2 x e R LK BAHH ST SC S BJ HB BL G SI CET

37 Permasalahan B Zero gradient: ?

38 S 2 -S (x)(I(x) - I0(x)) hy Permasalahan B’
Aperture problem (mis. Image datar): -S (x)(I(x) - I0(x)) x e R hy S x e R LK BAHH ST SC S BJ HB BL G SI CET

39 Permasalahan B’ No gradient along one direction: ?

40 Jawaban problem A & B Possible solutions: Manual intervention LK BAHH
ST SC S BJ HB BL G SI CET

41 Jawaban problem A & B Possible solutions: Manual intervention
Zero motion default LK BAHH ST SC S BJ HB BL G SI CET

42 Jawaban problem A & B Possible solutions: Manual intervention
Zero motion default Coefficient “dampening” LK BAHH ST SC S BJ HB BL G SI CET

43 Jawaban problem A & B Possible solutions: Manual intervention
Zero motion default Coefficient “dampening” Reliance on good features LK BAHH ST SC S BJ HB BL G SI CET

44 Jawaban problem A & B Possible solutions: Manual intervention
Zero motion default Coefficient “dampening” Reliance on good features Temporal filtering LK BAHH ST SC S BJ HB BL G SI CET

45 Jawaban problem A & B Possible solutions: Manual intervention
Zero motion default Coefficient “dampening” Reliance on good features Temporal filtering Spatial interpolation / hierarchical estimation LK BAHH ST SC S BJ HB BL G SI CET

46 Jawaban problem A & B Possible solutions: Manual intervention
Zero motion default Coefficient “dampening” Reliance on good features Temporal filtering Spatial interpolation / hierarchical estimation Higher-order terms LK BAHH ST SC S BJ HB BL G SI CET

47 Kembali lagi: Rumus Original
[ ] E ( h ) = I ( x + h ) - I0 ( x ) 2 x e R

48 S [ ] E ( h ) = I ( x + h ) - I0 ( x ) 2 Rumus Original
Transformations/warping of image S [ ] E ( h ) = I ( x + h ) - I0 ( x ) 2 x e R Translations: LK BAHH ST SC S BJ HB BL G SI CET

49 Bagaimana bila ada gerakan(motion) tipe lain?
Permasalahan C Bagaimana bila ada gerakan(motion) tipe lain?

50 S [ ] E ( A, h ) = I ( Ax + h ) - I0 ( x ) 2 Generalisasi 2a
Transformations/warping of image S [ ] E ( A, h ) = I ( Ax + h ) - I0 ( x ) 2 x e R Affine: LK BAHH ST SC S BJ HB BL G SI CET

51 Generalisasi 2a Affine:

52 S [ ] E ( A ) = I ( A x ) - I0 ( x ) 2 Generalisasi 2b
Transformations/warping of image S [ ] E ( A ) = I ( A x ) - I0 ( x ) 2 x e R Planar perspective: LK BAHH ST SC S BJ HB BL G SI CET

53 Generalisasi 2b Affine + Planar perspective:

54 Other parametrized transformations
Generalisasi 2c Transformations/warping of image S [ ] E ( h ) = I ( f(x, h) ) - I0 ( x ) 2 x e R Other parametrized transformations LK BAHH ST SC S BJ HB BL G SI CET

55 Other parametrized transformations
Generalisasi 2c Other parametrized transformations

56 S I0’(x)2 -S I0’(x)(I(x) - I0(x)) h ~ -(JTJ)-1 J (I(f(x,h)) - I0(x)) h
Permasalahan B” -S I0’(x)(I(x) - I0(x)) x e R h S I0’(x)2 Generalized aperture problem: ~ -(JTJ)-1 J (I(f(x,h)) - I0(x)) h LK BAHH ST SC S BJ HB BL G SI CET

57 Permasalahan B” Generalized aperture problem: ?

58 Rumus Original S [ ] E ( h ) = I ( x + h ) - I0 ( x ) 2 x e R

59 S [ ] E ( h ) = I ( x + h ) - I0 ( x ) 2 Rumus Original Image type
Grayscale images LK BAHH ST SC S BJ HB BL G SI CET

60 S || || E ( h ) = I ( x + h ) - I0 ( x ) 2 Generalisasi 3 Image type
Color images LK BAHH ST SC S BJ HB BL G SI CET

61 Rumus Original S [ ] E ( h ) = I ( x + h ) - I0 ( x ) 2 x e R

62 S [ ] E ( h ) = I ( x + h ) - I0 ( x ) 2 Rumus Original
Anggapan pixel punya konstan brightness (Constancy assumption) S [ ] E ( h ) = I ( x + h ) - I0 ( x ) 2 x e R Brightness constancy LK BAHH ST SC S BJ HB BL G SI CET

63 Bagaimana bila iluminasi cahaya bervariasi?
Permasalahan C Bagaimana bila iluminasi cahaya bervariasi?

64 Linear brightness constancy
Generalisasi 4a Constancy assumption S [ ] E ( h,a,b ) = I ( x + h ) - aI0 ( x )+b 2 x e R Linear brightness constancy LK BAHH ST SC S BJ HB BL G SI CET

65 Generalisasi 4a

66 Illumination subspace constancy
Generalisasi 4b Constancy assumption S [ ] E ( h,l ) = I ( x + h ) - lTB ( x ) 2 x e R Illumination subspace constancy LK BAHH ST SC S BJ HB BL G SI CET

67 Bagaimana bila texture berubah?
Permasalahan C’ Bagaimana bila texture berubah?

68 Texture subspace constancy
Generalisasi 4c Constancy assumption S [ ] E ( h,l ) = I ( x + h ) - lTB ( x ) 2 x e R Texture subspace constancy LK BAHH ST SC S BJ HB BL G SI CET

69 Permasalahan D Jelas proses konvergensi menjadi lambat bila jumlah #parameters bertambah !!!

70 Jawaban problem D Percepat konvergensi dengan:
Coarse-to-fine, filtering, interpolation, etc. LK BAHH ST SC S BJ HB BL G SI CET

71 Jawaban problem D Percepat konvergensi dengan:
Coarse-to-fine, filtering, interpolation, etc. Selective parametrization LK BAHH ST SC S BJ HB BL G SI CET

72 Jawaban problem D Percepat konvergensi dengan:
Coarse-to-fine, filtering, interpolation, etc. Selective parametrization Offline precomputation LK BAHH ST SC S BJ HB BL G SI CET

73 Jawaban problem D Percepat konvergensi dengan:
Coarse-to-fine, filtering, interpolation, etc. Selective parametrization Offline precomputation Difference decomposition LK BAHH ST SC S BJ HB BL G SI CET

74 Jawaban problem D Difference decomposition

75 Jawaban problem D Difference decomposition

76 Jawaban problem D Percepat konvergensi dengan:
Coarse-to-fine, filtering, interpolation, etc. Selective parametrization Offline precomputation Difference decomposition Improvements in gradient descent LK BAHH ST SC S BJ HB BL G SI CET

77 Jawaban problem D Percepat konvergensi dengan:
Coarse-to-fine, filtering, interpolation, etc. Selective parametrization Offline precomputation Difference decomposition Improvements in gradient descent Multiple estimates of spatial derivatives LK BAHH ST SC S BJ HB BL G SI CET

78 Jawaban problem D Multiple estimates / state-space sampling

79 Generalisasi metode Lucas-Kanade
Modifikasi yg. Dibuat selama ini adalah: S [ ] I ( x + h ) - I0 ( x ) 2 x e R

80 S [ ] I0 E ( h ) = I ( x + h ) - ( x ) 2 Rumus Original Error norm
Squared difference: LK BAHH ST SC S BJ HB BL G SI CET

81 Permasalahan dengan ourliers? >> Gunakan robust norm
Permasalahan E Permasalahan dengan ourliers? >> Gunakan robust norm

82 S ( ) E ( h ) = r I ( x + h ) - I0 ( x ) Generalisasi 5a Error norm
Robust error norm: LK BAHH ST SC S BJ HB BL G SI CET

83 Rumus Original S [ ] E ( h ) = I ( x + h ) - I0 ( x ) 2 x e R

84 S [ ] E ( h ) = I ( x + h ) - I0 ( x ) 2 Rumus Original
Image region / pixel weighting S [ ] E ( h ) = I ( x + h ) - I0 ( x ) 2 x e R Rectangular: LK BAHH ST SC S BJ HB BL G SI CET

85 Bagaimana bila background terjadi clutter (bergoyang)?
Permasalahan E’ Bagaimana bila background terjadi clutter (bergoyang)?

86 S [ ] I0 E ( h ) = I ( x + h ) - ( x ) 2 Generalisasi 6a
Image region / pixel weighting S [ I0 ] E ( h ) = I ( x + h ) - ( x ) 2 x e R Irregular: LK BAHH ST SC S BJ HB BL G SI CET

87 Bagaimana bila objek terhalang (foreground occlusion)?
Permasalahan E” Bagaimana bila objek terhalang (foreground occlusion)?

88 S [ ] I0 E ( h ) = I ( x + h ) - ( x ) w(x) 2 Generalisasi 6b
Image region / pixel weighting S [ I0 ] E ( h ) = I ( x + h ) - ( x ) w(x) 2 x e R Weighted sum: LK BAHH ST SC S BJ HB BL G SI CET

89 Generalisasi metode Lucas-Kanade
Modifikasi: S [ ] I ( x + h ) - I0 ( x ) 2 x e R

90 S [ ] E ( h ) = I ( x + h ) - I0 ( x ) 2 Generalisasi 6c
Image region / pixel weighting S [ ] E ( h ) = I ( x + h ) - I0 ( x ) 2 x e R Sampled: LK BAHH ST SC S BJ HB BL G SI CET

91 Generalisasi metode Lucas-Kanade: Ringkasan
[ ] E ( h ) = I ( x + h ) - I0 ( x ) 2 x e R S ( ) E ( h ) = r I ( f(x, h) ) - lB ( x ) w(x) x e R

92 Ringkasan Generalisasi L&K ? Dimension of image
Image transformations / motion models Pixel type Constancy assumption Error norm Image mask L&K ? Y n

93 Ringkasan Common problems: L&K ? Local minima Aperture effect
Illumination changes Convergence issues Outliers and occlusions L&K ? Y maybe n

94 Ringkasan Penanganan aperture effect: L&K ? Manual intervention
Zero motion default Coefficient “dampening” Elimination of poor textures Temporal filtering Spatial interpolation / hierarchical Higher-order terms L&K ? n Y