Detecting and eliminating covariate shifts in data for a more robust HDD failure prediction

K. Lukyanov   ; M. Drobyshevskiy   ; D. Turdakov

Zapiski Nauchnykh Seminarov POMI, Investigations on applied mathematics and informatics. Part IV, Tome 540 (2024), pp. 148-161

Résumé

Prediction of HDD failures has garnered significant attention in research, yet the persistence of covariate shifts in data remains a practical challenge. In this work we introduce a novel approach to training covariate shift detection models without the need for additional real data or artificial shift modeling. Moreover, we propose a comprehensive methodology integrating shift detection, administrator alerts, shift elimination, and HDD failure prediction. Experimental results demonstrate the viability of our real-world implementation.

Bibliographie

[1] P.P. Shinde and S. Shah, “A review of machine learning and deep learning applications”, 2018 Fourth International Conference on Computing Communication Control and Automation (ICCUBEA), 2018, 1–6

[2] M. Sugiyama and A.J. Storkey, “Mixture regression for covariate shift”, Advances in Neural Information Processing Systems, 19, 2006

[3] A. Bathelt, N.L. Ricker, and M. Jelali, “Revision of the Tennessee Eastman process model”, IFAC-Pap, 48 (2015)

[4] M. Kravchik and A. Shabtai, “Efficient cyber attack detection in industrial control systems using lightweight neural networks and PCA”, IEEE Trans. Dependable Secure Comput., 19 (2021)

[5] N.L. Ricker, “Decentralized control of the Tennessee Eastman challenge process”, J. Process Control, 6 (1996)

[6] S. Yin, S.X. Ding, A. Haghani, H. Hao, and P. Zhang, “A comparison study of basic data-driven fault diagnosis and process monitoring methods on the benchmark Tennessee Eastman process”, J. Process Control, 22 (2012) | DOI | MR

[7] G. Bontempi, S. Ben Taieb, and Y.A. Le Borgne, “Machine learning strategies for time series forecasting”, Business Intelligence: Second European Summer School, eBISS 2012, Tutorial Lectures 2 (Brussels, Belgium, July 15-21, 2012), 2013 | Zbl

[8] Z. Tao, Q. Xu, X. Liu, and J. Liu, “An integrated approach implementing sliding window and DTW distance for time series forecasting tasks”, Appl. Intell., 2023

[9] E.G. Silva, P.S.G. De Mattos Neto, and G.D.C. Cavalcanti, “A dynamic predictor selection method based on recent temporal windows for time series forecasting”, IEEE Access, 9 (2021) | MR

[10] M. Sugiyama and M. Kawanabe, Machine Learning in Non-Stationary Environments: Introduction to Covariate Shift Adaptation, MIT Press, 2012

[11] M.D. Prieto, G. Cirrincione, A.G. Espinosa, J.A. Ortega, and H. Henao, “Bearing fault detection by a novel condition-monitoring scheme based on statistical-time features and neural networks”, IEEE Trans. Ind. Electron., 60 (2012) | MR | Zbl

[12] A.S. Raj and N. Murali, “Early classification of bearing faults using morphological operators and fuzzy inference”, IEEE Trans. Ind. Electron., 60 (2012)

[13] W.-Y. Lin, Y.-H. Hu, and C.-F. Tsai, “Machine learning in financial crisis prediction: a survey”, IEEE Trans. Syst. Man Cybern. C, 42 (2011)

[14] K. Bluwstein, M. Buckmann, A. Joseph, S. Kapadia, and Ö. Śim{ş}ek, “Credit growth, the yield curve and financial crisis prediction: Evidence from a machine learning approach”, J. Int. Econ., 2023

[15] J. Uthayakumar, N. Metawa, K. Shankar, and S.K. Lakshmanaprabu, “Intelligent hybrid model for financial crisis prediction using machine learning techniques”, Inf. Syst. E-Bus. Manag., 18 (2020) | DOI

[16] Z. Wang, X. Su, and Z. Ding, “Long-term traffic prediction based on LSTM encoder-decoder architecture”, IEEE Trans. Intell. Transp. Syst., 22 (2020)

[17] W. Wang and M. Carr, “A stochastic filtering-based data-driven approach for residual life prediction and condition-based maintenance decision-making support”, 2010 Prognostics and System Health Management Conference, 2010, 1–10

[18] P. Lall, R. Lowe, and K. Goebel, “Extended Kalman filter models and resistance spectroscopy for prognostication and health monitoring of lead-free electronics under vibration”, IEEE Trans. Reliab., 61 (2012)

[19] S. Oeyen, K. Vermeulen, D. Benoit, L. Annemans, and J. Decruyenaere, “Development of a prediction model for long-term quality of life in critically ill patients”, J. Crit. Care, 43 (2018) | DOI

[20] G.F. Hughes, J.F. Murray, K. Kreutz-Delgado, and C. Elkan, “Improved disk-drive failure warnings”, IEEE Trans. Reliab., 51 (2002) | DOI

[21] V. Chandola, A. Banerjee, and V. Kumar, “Anomaly detection: A survey”, ACM Comput. Surv. (CSUR), 41 (2009) | DOI

[22] R. Chalapathy and S. Chawla, Deep learning for anomaly detection: A survey, 2019, arXiv: 1901.03407

[23] D. Samariya and A. Thakkar, “A comprehensive survey of anomaly detection algorithms”, Ann. Data Sci., 10 (2023)

[24] F. Salfner, M. Lenk, and M. Malek, “A survey of online failure prediction methods”, ACM Comput. Surv. (CSUR), 42 (2010) | DOI

[25] J.F. Murray, G.F. Hughes, and K. Kreutz-Delgado, “Hard drive failure prediction using non-parametric statistical methods”, Proceedings of ICANN/ICONIP, 2003

[26] Y. Wang, E.W.M. Ma, T.W.S. Chow, and K.-L. Tsui, “A two-step parametric method for failure prediction in hard disk drives”, IEEE Trans. Ind. Inform., 10 (2013)

[27] L.P. Queiroz, F.C.M. Rodrigues, J.P.P. Gomes, F.T. Brito, I.C. Chaves, M.R.P. Paula, M.R. Salvador, and J.C. Machado, “A fault detection method for hard disk drives based on mixture of Gaussians and nonparametric statistics”, IEEE Trans. Ind. Inform., 13 (2016)

[28] N. Georgoulopoulos, A. Hatzopoulos, K. Karamitsios, I.M. Tabakis, K. Kotrotsios, and A.I. Metsai, “A survey on hardware failure prediction of servers using machine learning and deep learning”, 2021 10th International Conference on Modern Circuits and Systems Technologies (MOCAST), 2021, 1–5

[29] J. Li, R.J. Stones, G. Wang, X. Liu, Z. Li, and M. Xu, “Hard drive failure prediction using decision trees”, Reliab. Eng. Syst. Saf, 164 (2017)

[30] J. Li, X. Ji, Y. Jia, B. Zhu, G. Wang, Z. Li, and X. Liu, “Hard drive failure prediction using classification and regression trees”, 2014 44th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, 2014, 383–394

[31] Q. Li, H. Li, and K. Zhang, “Prediction of HDD failures by ensemble learning”, 2019 IEEE 10th International Conference on Software Engineering and Service Science (ICSESS), 2019, 237–240

[32] Y. Ding, Y. Zhai, Y. Zhai, and J. Zhao, “Explore deep auto-coder and big data learning to hard drive failure prediction: a two-level semi-supervised model”, Connect. Sci., 34 (2022)

[33] R. Pinciroli, L. Yang, J. Alter, and E. Smirni, “Lifespan and failures of SSDs and HDDs: Similarities, differences, and prediction models”, IEEE Trans. Dependable Secure Comput., 2021

[34] S. Zahariah and H. Midi, “Minimum regularized covariance determinant and principal component analysis-based method for the identification of high leverage points in high dimensional sparse data”, J. Appl. Stat., 50 (2023) | DOI | MR | Zbl

[35] Y. Wang, K.L. Tsui, E.W.M. Ma, and M. Pecht, “A fusion approach for anomaly detection in hard disk drives”, Proceedings of the IEEE 2012 Prognostics and System Health Management Conference (PHM-2012 Beijing), 2012, 1–5

[36] S.M. Djurasevic, U.M. Pesovic, and B.S. Djordjevic, “Anomaly Detection Model for Predicting Hard Disk Drive Failures”, Appl. Artif. Intell., 35 (2021) | DOI

[37] M.W. Pozen, R.B. D'Agostino, H.P. Selker, P.A. Sytkowski, and W.B. Hood Jr, “A predictive instrument to improve coronary-care-unit admission practices in acute ischemic heart disease: a prospective multicenter clinical trial”, N. Engl. J. Med., 310 (1984) | DOI

[38] G. Ke, Q. Meng, T. Finley, T. Wang, W. Chen, W. Ma, Q. Ye, and T.-Y. Liu, “LightGBM: A highly efficient gradient boosting decision tree”, Advances in Neural Information Processing Systems, 30, 2017

[39] F. Rosenblatt, The perceptron, a perceiving and recognizing automaton. Project Para, Cornell Aeronautical Laboratory, 1957 | MR