Image Registration & Tracking dengan Metode Lucas & Kanade Sumber: Forsyth & Ponce Chap. 19, 20 Tomashi & Kanade: Good Feature to Track

Feature Lucas-Kanade(LK) Extraksi feature dengan metode LK ini adalah sangat populer dalam aplikasi computer vision. Feature diekstraksi dengan mengambil informasi gradient image. Selanjutnya feature ini bisa dimanfaatkan untuk Image registration, yg. Selanjutnya diugnakan utk. tracking, recognition, dan lain-lain Pemilihan feature image yang tepat adalah sangat menentukan keberhasilan proses recognition, tracking, etc.

Sejarah Perkembangan LK Lucas & Kanade (IUW 1981) BAHH ST S BJ HB BL G SI CET SC Bergen, Anandan, Hanna, Hingorani (ECCV 1992) Shi & Tomasi (CVPR 1994) Szeliski & Coughlan (CVPR 1994) Szeliski (WACV 1994) Black & Jepson (ECCV 1996) Hager & Belhumeur (CVPR 1996) Bainbridge-Smith & Lane (IVC 1997) Gleicher (CVPR 1997) Sclaroff & Isidoro (ICCV 1998) Cootes, Edwards, & Taylor (ECCV 1998) LK

Image Registration

Penerapan metode LK

Penerapan pada aplikasi: Stereo LK BAHH ST S BJ HB BL G SI CET SC

Penerapan pada aplikasi: Stereo Dense optic flow LK BAHH ST SC S BJ HB BL G SI CET

Penerapan pada aplikasi: Stereo Dense optic flow Image mosaics LK BAHH ST SC S BJ HB BL G SI CET

Penerapan pada aplikasi: Stereo Dense optic flow Image mosaics Tracking LK BAHH ST SC S BJ HB BL G SI CET

Penerapan pada aplikasi: Stereo Dense optic flow Image mosaics Tracking Recognition ? LK BAHH ST SC S BJ HB BL G SI CET

Derivasi Rumusan Lucas & Kanade #1

rumusan L&K 1 I0(x)

rumusan L&K 1 I0(x+h) h I0(x)

rumusan L&K 1 h I0(x) I(x)

rumusan L&K 1 h I0(x) I(x)

rumusan L&K 1 I0(x) I(x) R

rumusan L&K 1 I0(x) I(x)

rumusan L&K 1 h0 I0(x) I(x)

rumusan L&K 1 I0(x+h0) I(x)

rumusan L&K 1 I0(x+h1) I(x)

rumusan L&K 1 I0(x+hk) I(x)

rumusan L&K 1 I0(x+hf) I(x)

Derivasi Rumusan Lucas & Kanade #2

E(h) S [ I(x) - I0(x) - hI0’(x) ]2 rumusan L&K 2 Sum-of-squared-difference (SSD) error E(h) = S [ I(x) - I0(x+h) ]2 x e R E(h) S [ I(x) - I0(x) - hI0’(x) ]2 x e R

S I0’(x)2 S 2[I0’(x)(I(x) - I0(x) ) - hI0’(x)2] = 0 rumusan L&K 2 S 2[I0’(x)(I(x) - I0(x) ) - hI0’(x)2] x e R = 0 S I0’(x)(I(x) - I0(x)) x e R h S I0’(x)2

S S w(x) S I0’(x)2 w(x)[I(x) - I0(x)] I0’(x) h S I0’(x)[I(x) - I0(x)] Perbandingan h w(x)[I(x) - I0(x)] S w(x) x S I0’(x) S I0’(x)[I(x) - I0(x)] h S I0’(x)2 x

S S w(x) S I0’(x)2 w(x)[I(x) - I0(x)] I0’(x) h S I0’(x)[I(x) - I0(x)] Perbandingan w(x)[I(x) - I0(x)] S I0’(x) x h S w(x) x S I0’(x)[I(x) - I0(x)] x h S I0’(x)2 x

Generalisasi metode Lucas-Kanade

Rumus Original S [ ] E ( h ) = I ( x + h ) - I0 ( x ) 2 x e R

S [ ] E ( h ) = I ( x + h ) - I0 ( x ) 2 Rumus Original Dimension of image S [ ] E ( h ) = I ( x + h ) - I0 ( x ) 2 x e R 1-dimensional LK BAHH ST S BJ HB BL G SI CET SC

S [ ] E ( h ) = I ( x + h ) - I0 ( x ) 2 Generalisasi 1a Dimension of image S [ ] E ( h ) = I ( x + h ) - I0 ( x ) 2 x e R 2D: LK BAHH ST S BJ HB BL G SI CET SC

S [ ] E ( h ) = I ( x + h ) - I0 ( x ) 2 Generalisasi 1b Dimension of image S [ ] E ( h ) = I ( x + h ) - I0 ( x ) 2 x e R Homogeneous 2D: LK BAHH ST SC S BJ HB BL G SI CET

Apakah iterasi bisa konvergen? Permasalahan A Apakah iterasi bisa konvergen? LK BAHH ST S BJ HB BL G SI CET SC

Permasalahan A Local minima:

Permasalahan A Local minima:

S I0’(x)2 -S I0’(x)(I(x) - I0(x)) h Permasalahan B Zero gradient: x e R h is undefined if S I0’(x)2 is zero x e R S I0’(x)2 x e R LK BAHH ST SC S BJ HB BL G SI CET

Permasalahan B Zero gradient: ?

S 2 -S (x)(I(x) - I0(x)) hy Permasalahan B’ Aperture problem (mis. Image datar): -S (x)(I(x) - I0(x)) x e R hy S 2 x e R LK BAHH ST SC S BJ HB BL G SI CET

Permasalahan B’ No gradient along one direction: ?

Jawaban problem A & B Possible solutions: Manual intervention LK BAHH ST SC S BJ HB BL G SI CET

Jawaban problem A & B Possible solutions: Manual intervention Zero motion default LK BAHH ST SC S BJ HB BL G SI CET

Jawaban problem A & B Possible solutions: Manual intervention Zero motion default Coefficient “dampening” LK BAHH ST SC S BJ HB BL G SI CET

Jawaban problem A & B Possible solutions: Manual intervention Zero motion default Coefficient “dampening” Reliance on good features LK BAHH ST SC S BJ HB BL G SI CET

Jawaban problem A & B Possible solutions: Manual intervention Zero motion default Coefficient “dampening” Reliance on good features Temporal filtering LK BAHH ST SC S BJ HB BL G SI CET

Jawaban problem A & B Possible solutions: Manual intervention Zero motion default Coefficient “dampening” Reliance on good features Temporal filtering Spatial interpolation / hierarchical estimation LK BAHH ST SC S BJ HB BL G SI CET

Jawaban problem A & B Possible solutions: Manual intervention Zero motion default Coefficient “dampening” Reliance on good features Temporal filtering Spatial interpolation / hierarchical estimation Higher-order terms LK BAHH ST SC S BJ HB BL G SI CET

Kembali lagi: Rumus Original [ ] E ( h ) = I ( x + h ) - I0 ( x ) 2 x e R

S [ ] E ( h ) = I ( x + h ) - I0 ( x ) 2 Rumus Original Transformations/warping of image S [ ] E ( h ) = I ( x + h ) - I0 ( x ) 2 x e R Translations: LK BAHH ST SC S BJ HB BL G SI CET

Bagaimana bila ada gerakan(motion) tipe lain? Permasalahan C Bagaimana bila ada gerakan(motion) tipe lain?

S [ ] E ( A, h ) = I ( Ax + h ) - I0 ( x ) 2 Generalisasi 2a Transformations/warping of image S [ ] E ( A, h ) = I ( Ax + h ) - I0 ( x ) 2 x e R Affine: LK BAHH ST SC S BJ HB BL G SI CET

Generalisasi 2a Affine:

S [ ] E ( A ) = I ( A x ) - I0 ( x ) 2 Generalisasi 2b Transformations/warping of image S [ ] E ( A ) = I ( A x ) - I0 ( x ) 2 x e R Planar perspective: LK BAHH ST SC S BJ HB BL G SI CET

Generalisasi 2b Affine + Planar perspective:

Other parametrized transformations Generalisasi 2c Transformations/warping of image S [ ] E ( h ) = I ( f(x, h) ) - I0 ( x ) 2 x e R Other parametrized transformations LK BAHH ST SC S BJ HB BL G SI CET

Other parametrized transformations Generalisasi 2c Other parametrized transformations

S I0’(x)2 -S I0’(x)(I(x) - I0(x)) h ~ -(JTJ)-1 J (I(f(x,h)) - I0(x)) h Permasalahan B” -S I0’(x)(I(x) - I0(x)) x e R h S I0’(x)2 Generalized aperture problem: ~ -(JTJ)-1 J (I(f(x,h)) - I0(x)) h LK BAHH ST SC S BJ HB BL G SI CET

Permasalahan B” Generalized aperture problem: ?

Rumus Original S [ ] E ( h ) = I ( x + h ) - I0 ( x ) 2 x e R

S [ ] E ( h ) = I ( x + h ) - I0 ( x ) 2 Rumus Original Image type Grayscale images LK BAHH ST SC S BJ HB BL G SI CET

S || || E ( h ) = I ( x + h ) - I0 ( x ) 2 Generalisasi 3 Image type Color images LK BAHH ST SC S BJ HB BL G SI CET

Rumus Original S [ ] E ( h ) = I ( x + h ) - I0 ( x ) 2 x e R

S [ ] E ( h ) = I ( x + h ) - I0 ( x ) 2 Rumus Original Anggapan pixel punya konstan brightness (Constancy assumption) S [ ] E ( h ) = I ( x + h ) - I0 ( x ) 2 x e R Brightness constancy LK BAHH ST SC S BJ HB BL G SI CET

Bagaimana bila iluminasi cahaya bervariasi? Permasalahan C Bagaimana bila iluminasi cahaya bervariasi?

Linear brightness constancy Generalisasi 4a Constancy assumption S [ ] E ( h,a,b ) = I ( x + h ) - aI0 ( x )+b 2 x e R Linear brightness constancy LK BAHH ST SC S BJ HB BL G SI CET

Generalisasi 4a

Illumination subspace constancy Generalisasi 4b Constancy assumption S [ ] E ( h,l ) = I ( x + h ) - lTB ( x ) 2 x e R Illumination subspace constancy LK BAHH ST SC S BJ HB BL G SI CET

Bagaimana bila texture berubah? Permasalahan C’ Bagaimana bila texture berubah?

Texture subspace constancy Generalisasi 4c Constancy assumption S [ ] E ( h,l ) = I ( x + h ) - lTB ( x ) 2 x e R Texture subspace constancy LK BAHH ST SC S BJ HB BL G SI CET

Permasalahan D Jelas proses konvergensi menjadi lambat bila jumlah #parameters bertambah !!!

Jawaban problem D Percepat konvergensi dengan: Coarse-to-fine, filtering, interpolation, etc. LK BAHH ST SC S BJ HB BL G SI CET

Jawaban problem D Percepat konvergensi dengan: Coarse-to-fine, filtering, interpolation, etc. Selective parametrization LK BAHH ST SC S BJ HB BL G SI CET

Jawaban problem D Percepat konvergensi dengan: Coarse-to-fine, filtering, interpolation, etc. Selective parametrization Offline precomputation LK BAHH ST SC S BJ HB BL G SI CET

Jawaban problem D Percepat konvergensi dengan: Coarse-to-fine, filtering, interpolation, etc. Selective parametrization Offline precomputation Difference decomposition LK BAHH ST SC S BJ HB BL G SI CET

Jawaban problem D Difference decomposition

Jawaban problem D Difference decomposition

Jawaban problem D Percepat konvergensi dengan: Coarse-to-fine, filtering, interpolation, etc. Selective parametrization Offline precomputation Difference decomposition Improvements in gradient descent LK BAHH ST SC S BJ HB BL G SI CET

Jawaban problem D Percepat konvergensi dengan: Coarse-to-fine, filtering, interpolation, etc. Selective parametrization Offline precomputation Difference decomposition Improvements in gradient descent Multiple estimates of spatial derivatives LK BAHH ST SC S BJ HB BL G SI CET

Jawaban problem D Multiple estimates / state-space sampling

Generalisasi metode Lucas-Kanade Modifikasi yg. Dibuat selama ini adalah: S [ ] I ( x + h ) - I0 ( x ) 2 x e R

S [ ] I0 E ( h ) = I ( x + h ) - ( x ) 2 Rumus Original Error norm Squared difference: LK BAHH ST SC S BJ HB BL G SI CET

Permasalahan dengan ourliers? >> Gunakan robust norm Permasalahan E Permasalahan dengan ourliers? >> Gunakan robust norm

S ( ) E ( h ) = r I ( x + h ) - I0 ( x ) Generalisasi 5a Error norm Robust error norm: LK BAHH ST SC S BJ HB BL G SI CET

Rumus Original S [ ] E ( h ) = I ( x + h ) - I0 ( x ) 2 x e R

S [ ] E ( h ) = I ( x + h ) - I0 ( x ) 2 Rumus Original Image region / pixel weighting S [ ] E ( h ) = I ( x + h ) - I0 ( x ) 2 x e R Rectangular: LK BAHH ST SC S BJ HB BL G SI CET

Bagaimana bila background terjadi clutter (bergoyang)? Permasalahan E’ Bagaimana bila background terjadi clutter (bergoyang)?

S [ ] I0 E ( h ) = I ( x + h ) - ( x ) 2 Generalisasi 6a Image region / pixel weighting S [ I0 ] E ( h ) = I ( x + h ) - ( x ) 2 x e R Irregular: LK BAHH ST SC S BJ HB BL G SI CET

Bagaimana bila objek terhalang (foreground occlusion)? Permasalahan E” Bagaimana bila objek terhalang (foreground occlusion)?

S [ ] I0 E ( h ) = I ( x + h ) - ( x ) w(x) 2 Generalisasi 6b Image region / pixel weighting S [ I0 ] E ( h ) = I ( x + h ) - ( x ) w(x) 2 x e R Weighted sum: LK BAHH ST SC S BJ HB BL G SI CET

Generalisasi metode Lucas-Kanade Modifikasi: S [ ] I ( x + h ) - I0 ( x ) 2 x e R

S [ ] E ( h ) = I ( x + h ) - I0 ( x ) 2 Generalisasi 6c Image region / pixel weighting S [ ] E ( h ) = I ( x + h ) - I0 ( x ) 2 x e R Sampled: LK BAHH ST SC S BJ HB BL G SI CET

Generalisasi metode Lucas-Kanade: Ringkasan [ ] E ( h ) = I ( x + h ) - I0 ( x ) 2 x e R S ( ) E ( h ) = r I ( f(x, h) ) - lB ( x ) w(x) x e R

Ringkasan Generalisasi L&K ? Dimension of image Image transformations / motion models Pixel type Constancy assumption Error norm Image mask L&K ? Y n

Ringkasan Common problems: L&K ? Local minima Aperture effect Illumination changes Convergence issues Outliers and occlusions L&K ? Y maybe n

Ringkasan Penanganan aperture effect: L&K ? Manual intervention Zero motion default Coefficient “dampening” Elimination of poor textures Temporal filtering Spatial interpolation / hierarchical Higher-order terms L&K ? n Y