Variational relevance vector machine for classification and regression problems with multidimensional feature arrays

D. A. Kropotov

Žurnal vyčislitelʹnoj matematiki i matematičeskoj fiziki, Tome 51 (2011) no. 8, pp. 1541-1560

Résumé

Problems of classification and regression estimation in which objects are represented by multidimensional arrays of features are considered. Many practical statements can be reduced to such problems, for example, the popular approach to the description of images as a set of patches and a set of descriptors in each patch or the description of an object in the form of a set of distances from it to certain support objects selected based on a set of features. For solving problems concerning the objects thus described, a generalization of the relevance vector model is proposed. In this generalization, specific regularization coefficients are defined for each dimension of the multidimensional array of the object description; the resultant regularization coefficient for a given element in the multidimensional array is determined as a combination of the regularization coefficients for all the dimensions. The models with the sum and product used for such combinations are examined. Algorithms based on the variational approach are proposed for learning in these models. These algorithms enable one to find the so-called “sparse” solutions, that is, exclude from the consideration the irrelevant dimensions in the multidimensional array of the object description. Compared with the classical relevance vector model, the proposed approach makes it possible to reduce the number of adjustable parameters because a sum of all the dimensions is considered instead of their product. As a result, the method becomes more robust under overfitting in the case of small samples. This property and the sparseness of the resulting solutions in the proposed models are demonstrated experimentally, in particular, in the case of the known face identification database called Labeled Faces in the Wild.

Bibliographie

[1] Friedman J., Hastie T., Tibshirani R., The elements of statistical learning, Springer, New York, 2001 | MR

[2] Bishop C. M., Pattern recognition and machine learning, Springer, New York, 2006 | MR

[3] Vetrov D. P., Kropotov D. A., Osokin A. A., “Avtomaticheskoe opredelenie kolichestva komponent v EM-algoritme vosstanovleniya smesi normalnykh raspredelenii”, Zh. vychisl. matem. i matem. fiz., 50:4 (2010), 1–14 | MR

[4] Tipping M., “Sparse bayesian learning and the relevance vector machine”, J. Mach. Learn. Res., 1 (200), 211–244 | MR | Zbl

[5] Ahonen T., Hadid A., Pietrikaine M., “Face description with local binary patterns: Application to face recognition”, IEEE Trans. Pattern. Analys. Mach. Intel., 28:12 (2006), 2037–2041 | DOI

[6] Heikkila M., Pietikainen M., Schmid C., “Description of interest regions with center-symmetric local binary patterns”, Proc. 5th Indian Conf. Comput. Vis. Graph. Image Proc., 2006, 58–69 | DOI

[7] Yang J., Zhang D., Frangi A., Yang J., “Two-dimensional PCA: a new approach to appearance-based face representation and recognition”, IEEE Trans. Pattern. Analys. Mach. Intel., 26:1 (2004), 131–137 | DOI

[8] Zuo W., Zhang D., Wang K., “Bidirectional PCA with assembled matrix distance metric for image recognition”, IEEE Trans. Syst. Man Cyber. B, 36:4 (2006), 863–872 | DOI

[9] Xu D., Yan L., Zhang L. et al., Coupled subspaces analysis, Techn. Rept. No MSR-TR-2004-106, 2004

[10] Ye J., Janardan R., Li Q., “Two-dimensional linear discriminant analysis”, Advances Neural inform. Proc. Syst., 17 (2004)

[11] Li M., Yaun B., “2D-LDA: A statistical linear discriminant analysis for image matrix”, Pattern Rec. Letters, 26:5 (2005), 527–532 | DOI

[12] Yang W., Wang J., Ren M. et al., “Feature extraction based on laplacian bidirectional maximum margin criterion”, Pattern. Rec., 42:11 (2009), 2327–2334 | DOI | Zbl

[13] Wolf L., Jhuang H., Hazan T., “Modeling appearances with low-rank SVM”, Proc. Intern. Conf. Comput. Vis. Pattern. Res., 2007

[14] Boser B. E., Guyon I., Vapnik V., “A training algorithm for optimal margin classifiers”, Proc. Intern. Conf. Comput. Leartn. Theory, 1992, 144–152

[15] Tibshirani R., “Regression shrinkage and selection via the lasso”, J. Roy. Statist. Sco. B, 58:1 (1996), 267–288 | MR | Zbl

[16] MacKay D., “Bayesian interpolation”, Neural Comput., 4:3 (1992), 415–447 | DOI

[17] Williams P., “Bayesian regularization and pruning using a Laplace prior”, Neural Comput., 7:1 (1995), 117–143 | DOI

[18] Cawley G., Talbot N., “Gene selection in cancer classification using sparse logistic regression with Bayesian regularization”, Bioinformatics, 22 (2006), 2348–2355 | DOI

[19] Jordan M. I., Gharamani Z., Jaakkola T. S., Saul L. K., “An introduction to variational methods for graphical models”, Learning in Graphical Models, 1998, 105–162

[20] Bishop C., Tipping M., “Variational relevance vector machine”, Proc. Internat. Conf. Uncertainty Artif. Intel., 2000

[21] Minka T., “Expectation propagation for approximate Bayesian inference”, Proc. Internat. Conf. Uncertainty Artif. Intel., 2001, 362–369

[22] Jaakkola T., Jordan M., “Bayesian parameter estimation through variational methods”, Statist. Comput., 10 (2000), 25–37 | DOI

[23] Qi Y., Minka T., Picard R., Gharamani Z., “Predictive automatic relevance determination by expectation propagation”, Proc. Internat. Conf. Mach. Learn., 2004

[24] MacKay D., “Ensemble learning for hidden Markov models”, Techn. Rept. Dept. Phys. Univ. Cambridge, 1997

[25] Bishop C. M., “Bayesian PCA”, Advances Neural Inform. Proc. Sys., 11 (1999), 382–388

[26] Huang G. B., Ramesh M., Berg T., Learned-Miller E., Labeled faces in the Wild: A database for studying face recognition in unconstrained environments, Techn. Rept. No TR 07-49, 2007

[27] Ojala T., Pietikäinen M., Mäenpää T., “A generalized local binary pattern operator for multiresolution gray scale and rotation invariant texture classification”, Proc. Internat. Conf. Advances Pattern. Res., 2001

[28] Ojala T., Pietikäinen M., Mäenpää T., “Multiresolution gray-scale and rotation invariant texture classification with local binary patterns”, IEEE Trans. Pattern. Analys. Mach. Intel., 24:7 (2002), 971–987 | DOI

[29] Wolf L., Hassner T., Taigman Y., “Descriptor based methods in the wild”, Real-Life Images Workshop at ECCV, 2008