Analysis of correlation based dimension reduction methods

Shin, Y. J. ; Park, C. H.

International Journal of Applied Mathematics and Computer Science, Tome 21 (2011) no. 3, pp. 549-558.

Résumé

Dimension reduction is an important topic in data mining and machine learning. Especially dimension reduction combined with feature fusion is an effective preprocessing step when the data are described by multiple feature sets. Canonical Correlation Analysis (CCA) and Discriminative Canonical Correlation Analysis (DCCA) are feature fusion methods based on correlation. However, they are different in that DCCA is a supervised method utilizing class label information, while CCA is an unsupervised method. It has been shown that the classification performance of DCCA is superior to that of CCA due to the discriminative power using class label information. On the other hand, Linear Discriminant Analysis (LDA) is a supervised dimension reduction method and it is known as a special case of CCA. In this paper, we analyze the relationship between DCCA and LDA, showing that the projective directions by DCCA are equal to the ones obtained from LDA with respect to an orthogonal transformation. Using the relation with LDA, we propose a new method that can enhance the performance of DCCA. The experimental results show that the proposed method exhibits better classification performance than the original DCCA.

Détail

Keywords: canonical correlation analysis, dimension reduction, discriminative canonical correlation analysis, linear discriminant analysis
Mots-clés : analiza korelacyjna, redukcja wymiaru, liniowa analiza dyskryminacji

Bibliographie

[1] Anton, H. and Busby, R. (2003). Contemporary Linear Algebra, John Wiley and Sons, Denver, CO.

[2] Baudat, G. and Anouar, F. (2000). Generalized discriminant analysis using a kernel approach, Neural Computation 12(10): 2385-2404.

[3] Billings, S. and Lee, K. (2002). Nonlinear fisher discriminant analysis using a minimum squared error cost function and the orthogonal least squares algorithm, Neural Networks 15(2): 263-270.

[4] Chen, L., Liao, H., M. Ko, Lin, J. and Yu, G. (2000). A new LDA-based face recognition system which can solve the small sample size problem, Pattern Recognition 33(10): 1713-1726.

[5] Duda, R., Hart, P. and Stork, D. (2001). Pattern Classification, Wiley Interscience, New York, NY.

[6] Fukunaga, K. (1990). Introduction to Statistical Pattern Recognition, 2nd Edn., Academic Press, San Diego, CA.

[7] Fukunaga, K. and Mantock, J. (1983). Nonparametric discriminant analysis, IEEE Transactions on Pattern Analysis and Machine Intelligence 5(6): 671-678.

[8] Garthwaite, P. (1994). An interpretation of partial least squares, Journal of the American Statistical Society 89(425): 122-127.

[9] He, X. and Niyogi, P. (2003). Locality preserving projections, Proceedings of the Advances in Neural Information Processing Systems Conference, Vancouver, Canada, pp. 153-160.

[10] Hotelling, H. (1936). Relations between two sets of variates, Biometrika 28(3): 321-377.

[11] Hou, C., Nie, F., Zhang, C. and Wu, Y. (2009). Learning an orthogonal and smooth subspace for image classification, IEEE Signal Processing Letters 16(4): 303-306.

[12] Howland, P. and Park, H. (2004). Generalizing discriminant analysis using the generalized singular value decomposition, IEEE Transactions on Pattern Analysis and Machine Intelligence 26(8): 995-1006.

[13] Jolliffe, I. (1986). Principal Component Analysis, Springer, New York, NY.

[14] Nie, F., Xiang, S. and Zhang, C. (2007). Neighborhood minmax projections, Proceedings of the International Joint Conference on Artificial Intelligence, Hyderabad, India, pp. 993-998.

[15] Pardalos, P. and Hansen, P. (2008). Data Mining and Mathematical Programming, CRM Proceedings Lecture Notes, Vol. 45, American Mathematical Society, Montreal.

[16] Park, C. and Park, H. (2008). A comparison of generalized linear discriminant analysis algorithms, Pattern Recognition 41(3): 1083-1097.

[17] Roweis, S. T. and Saul, L. K. (2000). Nonlinear dimensionality reduction by locally linear embedding, Science 290(5500): 2323-2326.

[18] Sugiyama, M. (2006). Local fisher discriminant analysis for supervised dimensionality reduction, Proceedings of the IEEE International Conference on Machine Learning, Pittsburgh, PA, USA, pp. 905-912.

[19] Sun, Q., Zeng, S., Liu, Y., Heng, P. and Xia, D. (2005). A new method of feature fusion and its application in image recognition, Pattern Recognition 38(12): 2437-2448.

[20] Sun, T. and Chen, S. (2007). Class label versus sample label-based CCA, Applied Mathematics and Computation 185(1): 272-283.

[21] Sun, T., Chen, S., Yang, J. and Shi, P. (2008). A supervised combined feature extraction method for recognition, Procedings of the IEEE International Conference on Data Mining, Pisa, Italy, pp. 1043-1048.

[22] Tenenbaum, J. B., de Silva, V. and Langford, J. C. (2000). A global geometric framework for nonlinear dimensionality reduction, Science 290(5500): 2319-2323.

[23] Wegelin, J. (2000). A survey of partial least squares (PLS) methods, with emphasis on the two block case, Technical report, Department of Statistics, University of Washington, Seattle, WA.

[24] Yan, S., Xu, D., Zhang, B., Zhang, H.-J., Yang, Q. and Lin, S. (2007). Graph embedding and extensions: A general framework for dimensionlity reduction, IEEE Transactions on Pattern Analysis and Machine Intelligence 29(1): 40-51.

[25] Yang, J. and Yang, J.-Y. (2003). Why can LDA be performed in PCA transformed space?, Pattern Recognition 36(2): 563-566.

[26] Yang, J., Yang, J., Zhang, D. and Lu, J. (2003). Feature fusion: Parallel strategy vs. serial strategy, Pattern Recognition 36(6): 1369-1381.

[27] Ye, J. (2005). Characterization of a family of algorithms for generalized discriminant analysis on undersampled problems, Journal of Machine Learning Research 6(4): 483-502.

[28] Yu, H. and Yang, J. (2001). A direct LDA algorithm for highdimensional data-with application to face recognition, Pattern Recognition 34(10): 2067-2070.