Nonparametric methods for solving the data reconciliation problem

V. A. Garanin   ; K. K. Semenov

Zapiski Nauchnykh Seminarov POMI, Investigations on applied mathematics and informatics. Part IV, Tome 540 (2024), pp. 351-405

Résumé

Garanin V. A., Semenov K. K. Nonparametric methods for solving the data reconciliation problem. This work explores the current state of industrial data reconciliation, examining key approaches to addressing this problem in both classical and nonclassical contexts. The article provides a comprehensive overview of the main directions in this field and introduces novel contributions by the authors. These include approaches to nonparametric data reconciliation and a new method that revisits the analytical solution with the modern requirements to the restrictions on the reconciliation final results and at the current level of problem complexity. It offers closed-form expressions that enable the independent and targeted optimization of the industrial data reconciliation process for specific conditions —- a development not previously achieved. Finally, this paper aims to address the gap in the field of data reconciliation in Russian-language scientific literature.

Bibliographie

[1] M. J. Leibman, T. F. Edgar, L. S. Lasdon, “Efficient data reconciliation and estimation for dynamic processes using nonlinear programming techniques”, Computers chemical engineering, 16:10–11 (1992), 963-986 | DOI

[2] K. K. Semenov, G. N. Solopchenko, “Theoretical prerequisites for implementation of metrological self-tracking of measurement data analysis programs”, Measurement Techniques, 53:6 (2010), 592–599 | DOI

[3] K. K. Semenov, G. N. Solopchenko, “Combined method of metrological self-tracking of measurement data processing programs general problems of metrology and measurement technique”, Measurement Techniques, 54:4 (2011), 378–386 | DOI

[4] K. K. Semenov, Metrologicheskoe avtosoprovozhdenie programm vychislenii v informatsionno-izmeritelnykh sistemakh, Dissertatsiya na soiskanie uchenoi stepeni kandidata tekhnicheskikh nauk po spetsialnosti 05.11.16. Informatsionno-izmeritelnye i upravlyayuschie sistemy, Sankt-Peterburg, 2011

[5] D. R. Kuehn, H. Davidson, “Computer control”, Chem. Engng. Prog., 57 (1961), 44–47

[6] V. Vaclavek, “Studies on system engineering. II. On the application of the calculus of observations in calculations of chemical engineering balances”, Collection of Czechoslovak Chemical Communications, 34:2 (1969), 364–372 | DOI

[7] V. Vaclavek, M. Loučka, “Selection of measurements necessary to achieve multicomponent mass balances in chemical plant”, Chemical Engineering Science, 31:12 (1976), 1199–1205 | DOI

[8] R. L. Wiegel, “Advances in mineral processing material balances”, Canadian Metallurgical Quarterly, 11:2 (1972), 413–424 | DOI

[9] D. Hodouin, M. D. Everell, “A hierarchical procedure for adjustment and material balancing of mineral processes data”, International Journal of Mineral Processing, 7:2 (1980), 91–116 | DOI

[10] D. E. Simpson, V. R. Voller, M. G. Everett, “An efficient algorithm for mineral processing data adjustment”, International Journal of Mineral Processing, 31:1-2 (1991), 73–96 | DOI

[11] N. Heraud, D. Maquin, J. Ragot, “Multilinear balance equilibration: Application to a complex metallurgical process”, Mining, Metallurgy Exploration, 8:4 (1991), 197–204 | DOI

[12] K. J. Reid, et al., “A Survey of Material Balance Computer Packages in the Mineral Processing Industry”, Application of Computers and Operations Research in the Mineral Industry, Soc. of Min. Eng. of AIME, 1982, 41–62

[13] R. S. Mah, G. M. Stanley, D. M. Downing, “Reconcillation and rectification of process flow and inventory data”, Industrial Engineering Chemistry Process Design and Development, 15:1 (1976), 175–183 | DOI

[14] C. M. Crowe, Y. A. G. Campos, A. Hrymak, “Reconciliation of process flow rates by matrix projection. Part I: linear case”, AIChE Journal, 29:6 (1983), 881–888 | DOI

[15] C. L. E. Swartz, “Data reconciliation for generalized flowsheet applications”, The American chemical society Journal, 197 (1989), 58–64

[16] M. Sanchez, J. Romagnoli, “Use of orthogonal transformations in data classification-reconciliation”, Computers chemical engineering, 20:5 (1996), 483–493 | DOI

[17] D. L. Ripps, “Adjustment of experimental data”, Chem. Eng. Prog. Symp. Ser., 61:8 (1965), 8–13

[18] P. M. Reilly, R. E. Carpani, “Application of statistical theory of adjustment to material balances”, 13th Canadian Chemical Engineering Congress (Montreal, Quebec), 1963

[19] G. A. Almasy, T. Sztano, “Checking and Correction of Measurements on the Basis of Linear System Model”, Prob. Control Inform. Theory, 4 (1975), 57–69 | MR | Zbl

[20] S. Narasimhan, R.S.H. Mah, “Generalized Likelihood Ratio Method for Gross Error Identification”, AIChE Journal, 33 (1987), 1514–1521 | DOI | MR

[21] A. C. Tamhane, C. Jordache, R.S.H. Mah, “A Bayesian Approach to Gross Error Detection in Chemical Process Data. Part I: Model Development”, Chemometrics and Intel. Lab. Sys., 4 (1988), 33–45 | DOI

[22] H. Tong, C. M. Crowe, “Detection of Gross Errors in Data Reconciliation by Principal Component Analysis”, AIChE Journal, 41 (1995), 1712–1722 | DOI | MR

[23] J. A. Romagnoli, G. Stephanopoulos, “Rectification of Process Measurement Data in the Presence of Gross Errors”, Chem. Eng. Sci., 36 (1981), 1849–1863 | DOI

[24] R. W. Serth, W. A. Heenan, “Gross Error Detecting and Data Reconciliation in Steam-Metering Systems”, AIChE Journal, 30 (1986), 743–747

[25] J. Rosenberg, R. S. H. Mah, C. Jordache, “Evaluation of Schemes for Detecting and Identification of Gross Errors in Process Data”, Ind. Eng. Chem. Proc. Des. Dev, 26 (1987), 555–564

[26] C. Jordache, R. S. H. Mah, A. C. Tamhane, “Performance Studies of the Measurement Test for Detecting of Gross Error in Process Data”, AIChE Journal, 31 (1985), 1187–1201 | DOI

[27] D. K. Rollins, J. F. Davis, “Unbiased Estimation Technique for Identification of Gross Errors”, AIChE Journal, 38 (1992), 563–571 | DOI

[28] V. Hlavacek, “Analysis of a Complex Plant–Steady State and Transient Behavior I-Plant Data Estimation and Adjustment”, Computers Chem. Engng., 1 (1977), 75–81 | DOI

[29] R. S. H. Mah, “Design and Analysis of Performance Monitoring Systems”, Chemical Process Control II, Engineering Foundation, New York, 1982

[30] A. C. Tamhane, R. S. H. Mah, “Data Reconciliation and Gross Error Detection in Chemical Process Networks”, Technometrics, 27 (1985), 409–422 | DOI

[31] R. S. H. Mah, “Data Screening”, Foundations of Computer-Aided Process Operations, CACHE/Elsevier, Amsterdam, 1987, 67–94

[32] C. M. Crowe, “Data Reconciliation Progress and Challenges”, J. Proc. Cont, 6 (1996), 89–98 | DOI

[33] R. S. H. Mah, Chemical Process Structures, and Information Flows, Butterworths, Boston, 1990

[34] C. E. Bodington, Planning, Scheduling and Control Integration in Process Industries, McGraw-Hill, New York, 1995

[35] F. Madron, Process Plant Performance: Measurement and Data Processing for Optimization and Retrofits, Ellis Horwood Limited Co, Chichester, West Sussex, England, 1992

[36] V. V. Veverka, F. Madron, Material and Energy Balancing in Process Industries: From Microscopic Balances to Large Plants, Elsevier, Amsterdam, Netherlands, 1997

[37] J. C. Knepper, J. W. Gorman, “Statistical Analysis of Constrained Data Sets”, AIChE Journal, 26 (1980), 260–264 | DOI | MR

[38] H. I. Britt, R. H. Luecke, “The Estimation of Parameters in Nonlinear Implicit Models”, Technometrics, 15 (1973), 233–247 | DOI | Zbl

[39] A. Ruszczyński, “A linearization method for nonsmooth stochastic programming problems”, Mathematics of Operations Research, 12:1 (1987), 32–49 | DOI | MR | Zbl

[40] C. M. Crowe, “Reconciliation of process flow rates by matrix projection. Part II: The nonlinear case”, AIChE Journal, 32:4 (1986), 616–623 | DOI

[41] I. B. Tjoa, L. T. Biegler, “Simultaneous Strategies for Data Reconciliation and Gross Error Detection of Nonlinear Systems”, Computers Chem. Engng., 15 (1991), 679–690 | DOI

[42] V. Ravikumar, S. R. Singh, M. O. Garg, S. Narasimhan, “RAGEmA Software Tool for Data Reconciliation and Gross Error Detection”, Foundations of Computer-Aided Process Operations, CACHE/Elsevier, Amsterdam, 1994, 429–436

[43] C. De Mulder, et al., “An open software package for data reconciliation and gap filling in preparation of Water and Resource Recovery Facility Modeling”, Environmental Modelling Software, 107 (2018), 186–198 | DOI

[44] A. Farzi, A. Mehrabani-Zeinabad, R. B. Boozarjomehry, “Data reconciliation: Development of an object-oriented software tool”, Korean Journal of Chemical Engineering, 25:5 (2008), 955–965 | DOI

[45] H. L. Lam, et al., “Software tools overview: process integration, modelling and optimisation for energy saving and pollution reduction”, Asia-Pacific Journal of Chemical Engineering, 6:5 (2011), 696–712 | DOI

[46] F. Manenti, et al., “Adaptive data reconciliation coupling C++ and PRO/II and on-line application by the field”, Computer Aided Chemical Engineering, 28 (2010), 373–378 | DOI

[47] G. Buzzi-Ferraris, F. Manenti, “BzzMath: Library overview and recent advances in numerical methods”, Computer Aided Chemical Engineering, 30 (2012), 1312–1316 | DOI

[48] O. Cencic, “Nonlinear data reconciliation in material flow analysis with software STAN”, Sustainable Environment Research, 26:6 (2016), 291–298 | DOI

[49] T. Jiang, et al., “Strategy for improving data quality for a Kraft pulp mill recausticizing plant”, FOCAPO 2003: Proceedings Foundations Of Computer Aided Process Operations, 2003, 467–470

[50] J. Y. Yong, et al., “Data reconciliation for energy system flowsheets”, Computer Aided Chemical Engineering, 38 (2016), 2277–2282 | DOI

[51] M. M. Camara, et al., “Numerical aspects of data reconciliation in industrial applications”, Processes, 5:4 (2017), 56 | DOI

[52] G. M. Stanley, R. S. H. Mah, “Estimation of Flows and Temperatures in Process Networks”, AIChE Journal, 23 (1977), 642–650 | DOI

[53] S. Narasimhan, C. Jordache, Data reconciliation and gross error detection: An intelligent use of process data, Elsevier, 1999

[54] Y. Ramamurthi, P. B. Sistu, B. W. Bequette, “Control Relevant Dynamic Data Reconciliation and Parameter Estimation”, Computers Chem. Engng., 17:1 (1993), 41–59 | DOI

[55] I. W. Kim, M. J. Liebman, T. F. Edgar, “Robust Error in Variables Estimation using Nonlinear Programming Techniques”, AIChE Journal., 36 (1990), 985-993 | DOI

[56] I. W. Kim, M. J. Liebman, T. F. Edgar, “A Sequential Error in Variables Estimation Method for Nonlinear Dynamic Systems”, Computers Chem. Engng., 15 (1991), 663–670 | DOI

[57] F. Allgöwer, et al., “Nonlinear predictive control and moving horizon estimation-an introductory overview”, Advances in control. Highlights of ECC'99, 1999, 391–449

[58] W. R. Esposito, C. A. Floudas, “Global optimization for the parameter estimation of differential-algebraic systems”, Industrial Engineering Chemistry Research, 39:5 (2000), 1291–1310 | DOI | MR

[59] C. V. Rao, J. B. Rawlings, D. Q. Mayne, “Constrained state estimation for nonlinear discrete-time systems: Stability and moving horizon approximations”, IEEE transactions on automatic control, 48:2 (2003), 246–258 | DOI | MR | Zbl

[60] D. B. Özyurt, R. W. Pike, “Theory and practice of simultaneous data reconciliation and gross error detection for chemical processes”, Computers Chemical Engineering, 28:3 (2004), 381–402 | DOI

[61] D. M. Prata, et al., “Nonlinear dynamic data reconciliation and parameter estimation through particle swarm optimization: Application for an industrial polypropylene reactor”, Chemical Engineering Science, 64:18 (2009), 3953–3967 | DOI

[62] C. E. Llanos, M. C. Sanchez, R. A. Maronna, “Robust estimators for data reconciliation”, Industrial Engineering Chemistry Research, 54:18 (2015), 5096–5105 | DOI

[63] P. Groeneboom, G. Jongbloed, Nonparametric estimation under shape constraints., Cambridge Series in Statistical and Probabilistic Mathematics, 38, Cambridge University Press, 2014 | MR | Zbl

[64] J. L. Horowitz, S. Lee, “Nonparametric estimation and inference under shape restrictions”, Journal of Econometrics, 201:1 (2017), 108–126 | DOI | MR | Zbl

[65] K. Shariatmadar, M. Versteyhe, “Linear programming under p-box uncertainty model”, 7th International Conference on Control, Mechatronics and Automation, 2019, 84–89

[66] M. Bagajewicz, “Data reconciliation and instrumentation upgrade. Overview and challenges”, FOCAPO (Coral Springs, FL, USA), Foundations of Computer Aided Process Operations, 372, 2003

[67] M. Bagajewicz, M. Markowski, “Instrumentation design and upgrade using an unconstrained method with pure economical objectives”, Proceedings of FOCAPO, 2003

[68] M. Gala, M. J. Bagajewicz, “Rigorous methodology for the design and upgrade of sensor networks using cutsets”, Industrial Engineering Chemistry Research, 45:20 (2006), 6679–6686 | DOI

[69] P. R. Kotecha, M. Bhushan, R. D. Gudi, “Multi-objective optimization based robust sensor network design”, IFAC Proceedings Volumes, 43:5 (2010), 79–84 | DOI

[70] L. K. Reznik, G. N. Solopchenko, “Use of priori information on functional relations between measured quantities for improving accuracy of measurement”, Measurement, 3:3 (1985), 98–106 | DOI

[71] C. C. D. Pai, G. Fisher, “Application of Broyden's Method to Reconciliation of Nonlinearly Constrained Data”, AIChE Journal, 34:5 (1988), 873–876 | DOI

[72] S. J. Julier, J. K. Uhlmann, “New extension of the Kalman filter to nonlinear systems”, Signal processing, sensor fusion, and target recognition VI, Proc. SPIE, 3068, International Society for Optics and Photonics, 1997, 182–193 | DOI

[73] D. Wang, J. A. Romagnoli, “Robust data reconciliation based on a generalized objective function”, IFAC Proceedings Volumes, 35:1 (2002), 191–196

[74] F. R. Hampel, et al., Robust statistics: the approach based on influence functions, Wiley-Interscience, New York, 1986 | MR | Zbl

[75] V. A. Garanin, K. K. Semenov, “Semi-nonparametric approach for measured data reconciliation based on the Gram-Charlier series expansion”, Measurement: Sensors, 18 (2021), 100351 | DOI

[76] V. A. Garanin, K. K. Semenov, “Increasing measurement accuracy by nonparametric data reconciliation”, Measurement, 238 (2024), 115235 | DOI

[77] V. A. Garanin, K. K. Semenov, “Measured data reconciliation processed within nonlinear constraints models in cyber-physical systems”, Lecture Notes in Networks and Systems, 460, 2023, 78–95 | DOI

[78] V. A. Garanin, K. K. Semenov, “Estimating the accuracy increase during the measuring two quantities with linear dependence”, Lecture Notes in Networks and Systems, 95, 2020, 235–246 | DOI

[79] J. E. Kolassa, Series approximation methods in statistics, Lecture Notes in Statistics, 88, Springer Science Business Media, 2006 | MR | Zbl

[80] T. Jaki, R. W. West, “Maximum kernel likelihood estimation”, Journal of Computational and Graphical Statistics, 17:4 (2008), 976–993 | DOI | MR

[81] A. A. Efremov, V. N. Kozlov, “Metod sinteza lokalno dopustimykh ogranichennykh upravlenii dlya stabilizatsii programmnykh dvizhenii dinamicheskikh ob'ektov”, Informatsionno-upravlyayuschie sistemy, 2023, no. 4, 47–55

[82] A. A. Efremov, “Projection operator for solving generalized problems of program motions stabilization”, Informatika, telekommunikatsii i upravlenie, 16:4 (2023), 49–59