Geodesic
Extrapolation of the Bayesian classifier with an unknown support of the two-class mixture distribution
Trudy Matematicheskogo Instituta imeni V.A. Steklova, Tome 79 (2024) no. 6, pp. 991-1015 Cet article a éte moissonné depuis la source Math-Net.Ru

Voir la notice de l'article

This work introduces a method aimed at enhancing the reliability of the Bayesian classifier. The method involves augmenting the training dataset, which consists of a mixture of distributions from two original classes, with artificially generated observations from a third, ‘background’ class, uniformly distributed over a compact set that contains the unknown support of the original mixture. This modification allows the value of the discriminant function outside the support of the training data distribution to approach a prescribed level (in this case, zero). Adding a decision option for ‘Refusal to Classify’, triggered when the discriminant function takes sufficiently small values, results in a localized increase in classifier reliability. Specifically, this approach addresses several issues: it enables the rejection of data that differs significantly from the training data; facilitates the detection of anomalies in input data; and avoids decision-making in ‘boundary’ regions when separating classes. The paper provides a theoretical justification for the optimality of the proposed classifier. The practical utility of the method is demonstrated through classification tasks involving images and time series. Additionally, a methodology for identifying trusted regions is proposed. This methodology can be used to detect anomalous data, cases of parameter shifts in class distributions, and areas of overlap between the distributions of the original classes. Based on these trusted regions, quantitative metrics for classifier reliability and efficiency are introduced. Bibliography: 23 titles.
Keywords: machine learning, Bayesian classifier, trusted machine learning, interpretability, out-of-distribution (OOD), time series classification, rejection of classification, background class.
Mots-clés : image classification 
@article{RM_2024_79_6_a3,
     author = {K. S. Lukyanov and P. A. Yaskov and A. I. Perminov and A. P. Kovalenko and D. Y. Turdakov},
     title = {Extrapolation of the {Bayesian} classifier with an unknown support of the two-class mixture distribution},
     journal = {Trudy Matematicheskogo Instituta imeni V.A. Steklova},
     pages = {991--1015},
     year = {2024},
     volume = {79},
     number = {6},
     language = {en},
     url = {http://geodesic.mathdoc.fr/item/RM_2024_79_6_a3/}
}
TY  - JOUR
AU  - K. S. Lukyanov
AU  - P. A. Yaskov
AU  - A. I. Perminov
AU  - A. P. Kovalenko
AU  - D. Y. Turdakov
TI  - Extrapolation of the Bayesian classifier with an unknown support of the two-class mixture distribution
JO  - Trudy Matematicheskogo Instituta imeni V.A. Steklova
PY  - 2024
SP  - 991
EP  - 1015
VL  - 79
IS  - 6
UR  - http://geodesic.mathdoc.fr/item/RM_2024_79_6_a3/
LA  - en
ID  - RM_2024_79_6_a3
ER  - 
%0 Journal Article
%A K. S. Lukyanov
%A P. A. Yaskov
%A A. I. Perminov
%A A. P. Kovalenko
%A D. Y. Turdakov
%T Extrapolation of the Bayesian classifier with an unknown support of the two-class mixture distribution
%J Trudy Matematicheskogo Instituta imeni V.A. Steklova
%D 2024
%P 991-1015
%V 79
%N 6
%U http://geodesic.mathdoc.fr/item/RM_2024_79_6_a3/
%G en
%F RM_2024_79_6_a3
K. S. Lukyanov; P. A. Yaskov; A. I. Perminov; A. P. Kovalenko; D. Y. Turdakov. Extrapolation of the Bayesian classifier with an unknown support of the two-class mixture distribution. Trudy Matematicheskogo Instituta imeni V.A. Steklova, Tome 79 (2024) no. 6, pp. 991-1015. http://geodesic.mathdoc.fr/item/RM_2024_79_6_a3/

[1] A. Jishan and R. C. Green II, “Cost aware LSTM model for predicting hard disk drive failures based on extremely imbalanced S.M.A.R.T. sensors data”, Eng. Appl. Artif. Intell., 127 (2024), 107339, 11 pp. | DOI

[2] A. Caron, C. Hicks, and V. Mavroudis, A view on out-of-distribution identification from a statistical testing theory perspective, 2024, 8 pp., arXiv: 2405.03052

[3] Peng Cui and Jinjia Wang, “Out-of-distribution (OOD) detection based on deep learning: a review”, Electronics, 11:21 (2022), 3500, 19 pp. | DOI

[4] L. Devroye, L. Györfi, and G. Lugosi, A probabilistic theory of pattern recognition, Appl. Math. (N. Y.), 31, Reprint of the 1996 original, Springer-Verlag, New York, 2013, xvi+636 pp. | DOI | MR | Zbl

[5] S. M. Djurasevic, U. M. Pesovic, and B. S. Djordjevic, “Anomaly detection model for predicting hard disk drive failures”, Appl. Artif. Intell., 35:8 (2021), 549–566 | DOI

[6] A. Faragó and G. Lugosi, “Strong universal consistency of neural network classifiers”, IEEE Trans. Inform. Theory, 39:4 (1993), 1146–1151 | DOI | Zbl

[7] D. Hendrycks and K. Gimpel, A baseline for detecting misclassified and out-of-distribution examples in neural networks, 2016 (v1 – 2016), 12 pp., arXiv: 1610.02136

[8] J. Jithish, B. Alangot, N. Mahalingam, and Kiat Seng Yeo, “Distributed anomaly detection in smart grids: a federated learning-based approach”, IEEE Access, 11 (2023), 7157–7179 | DOI

[9] A. Klein, Backblaze: Hard drive data and stats https://www.backblaze.com/cloud-storage/resources/hard-drive-test-data

[10] Lingdong Kong, Shaoyuan Xie, Hanjiang Hu, Lai Xing Ng, B. Cottereau, and Wei Tsang Ooi, “Robodepth: Robust out-of-distribution depth estimation under corruptions”, Adv. Neural Inf. Process. Syst., 36 (2023), 1–45

[11] Bo Li, Peng Qi, Bo Liu, Shuai Di, Jingen Liu, Jiquan Pei, Jinfeng Yi, and Bowen Zhou, “Trustworthy AI: from principles to practices”, ACM Comput. Surveys, 55:9 (2023), 177, 46 pp. | DOI

[12] Jeremiah Zhe Liu, S. Padhy, Jie Ren, Zi Lin, Yeming Wen, G. Jerfel, Z. Nado, J. Snoek, D. Tran, and B. Lakshminarayanan, “A simple approach to improve single-model deep uncertainty via distance-awareness”, J. Mach. Learn. Res., 24 (2023), 42, 63 pp. | MR

[13] A. B. Nassif, M. Abu Talib, Q. Nasir, and F. M. Dakalbab, “Machine learning for anomaly detection: a systematic review”, IEEE Access, 9 (2021), 78658–78700 | DOI

[14] M. Perello-Nieto, T. D. M. E. S. Filho, M. Kull, and P. Flach, “Background check: a general technique to build more reliable and versatile classifiers”, 2016 IEEE 16th international conference on data mining (ICDM), IEEE, 2016, 1143–1148 | DOI

[15] R. Pinciroli, L. Yang, J. Alter, and E. Smirni, “Lifespan and failures of SSDs and HDDs: similarities, differences, and prediction models”, IEEE Trans. Depend. Secure Comput., 20:1 (2023), 256–272 | DOI

[16] K. Rasheed, A. Qayyum, M. Ghaly, A. Al-Fuqaha, A. Razi, and J. Qadir, “Explainable, trustworthy, and ethical machine learning for healthcare: a survey”, Comput. Biol. Med., 149 (2022), 106043, 23 pp. | DOI

[17] Boxin Wang, Weixin Chen, Hengzhi Pei, Chulin Xie, Mintong Kang, Chenhui Zhang, Chejian Xu, Zidi Xiong, R. Dutta, R. Schaeffer, Sang T. Truong, Simran Arora, M. Mazeika, D. Hendrycks, Zinan Lin, Yu Cheng, S. Koyejo, Dawn Song, and Bo Li, DecodingTrust: a comprehensive assessment of trustworthiness in GPT models, 2024 (v1 – 2023), 110 pp., arXiv: 2306.11698

[18] Qibo Yang, Xiaodong Jia, Xiang Li, Jianshe Feng, Wenzhe Li, and Jay Lee, “Evaluating feature selection and anomaly detection methods of hard drive failure prediction”, IEEE Trans. Reliab., 70:2 (2021), 749–760 | DOI

[19] Hang Yu, Weixu Liu, Jie Lu, Yimin Wen, Xiangfeng Luo, and Guangquan Zhang, “Detecting group concept drift from multiple data streams”, Pattern Recognition, 134 (2023), 109113, 11 pp. | DOI

[20] He Zhang, Bang Wu, Xingliang Yuan, Shirui Pan, Hanghang Tong, and Jian Pei, “Trustworthy graph neural networks: aspects, methods, and trends”, Proc. IEEE, 112:2 (2024), 97–139 | DOI

[21] Jing Zhang, Yuchao Dai, Mochu Xiang, Deng-Ping Fan, P. Moghadam, Mingyi He, C. Walder, Kaihao Zhang, M. Harandi, and N. Barnes, Dense uncertainty estimation, 2021, 15 pp., arXiv: 2110.06427

[22] Mingyu Zhang, Wenqiang Ge, Ruichun Tang, and Peishun Liu, “Hard disk failure prediction based on blending ensemble learning”, Appl. Sci., 13:5 (2023), 3288, 22 pp. | DOI

[23] Zhilin Zhao, Statistical methods for out-of-distribution detection, PhD thesis, Univ. Technology Sydney, 2023, 107 pp. | MR