Time Series Outlier Detection for Short-Term Electricity Load Demand Forecasting
Article Sidebar
Published:
Mar 5, 2019
Keywords:
Data Preprocessing Data Cleaning Outliers Detection Filtering Band Short-Term Load Forecasting and Neural Network
Main Article Content
Abstract
Forecasting of working-days’ electricity demand is vital for short-term planning. However, demand variations due to outliers can reduce the accuracy of forecasts. Therefore, a time series data cleaning technique is proposed to remove these disturbances of electricity data. First, holidays’ and bridging holidays’ data are replaced by Moving Average. The k-sliding window filtering band is proposed to detect the time series outliers and replace by forecasted regular load demand using Moving Average. Data from the Electricity Generating Authority of Thailand (EGAT) and a Neural Network (NN) model with six inputs and one output are used to demonstrate the performance of time window data cleaning process. The sample dataset contains data from 1stMay 2012 to 31stMay 2013 where May 2013 is used for testing. The Time-Window based data cleaning technique increases the performance of forecasting outcomes by 11.60% for non-holidays. Results from the proposed technique are compared with the results from the robust version of locally weighted smoothing (r-LOESS) and identified that the proposed technique is superior for taking results for non-holidays.
Article Details
How to Cite
Jeenanunta, C., Abeyrathna, K. D., Dilhani, M. H. M. R. S., Hnin, S. W., & Phyo, P. P. (2019). Time Series Outlier Detection for Short-Term Electricity Load Demand Forecasting. INTERNATIONAL SCIENTIFIC JOURNAL OF ENGINEERING AND TECHNOLOGY (ISJET), 2(1), 37–50. Retrieved from https://ph02.tci-thaijo.org/index.php/isjet/article/view/175908
Section
Research Article
เนื้อหาข้อมูล
References
[1] A. D. Papalexopoulos, and T. C. Hesterberg, “A regressionbased approach to short-term system load forecasting,” IEEE Transactions on Power Systems, vol. 5, no. 4, pp. 1535-1547, 1990.
[2] R. Sadownik, and E. P. Barbosa, “Shortterm forecasting of industrial electricity consumption in Brazil,” Journal of
Forecasting, vol. 18, no. 3, pp. 215-224, 1999.
[3] Y. Ohtsuka, and K. Kakamu, “SpaceTime Model versus VAR Model: Forecasting Electricity demand in Japan.” Journal
of Forecasting, vol. 32, no. 1, pp. 75-85, 2013.
[4] U. D. Caprio et al., “Short term load forecasting in electric power systems: a comparison of ARMA models and extended wiener filtering.” Journal of Forecasting, vol. 2, no. 1, pp. 59-76, 1983.
[5] G. J. Janacek, and L. Swift, Time series: forecasting, simulation, applications. Ellis horwood, 1993.
[6] A. K. Singh et al., “An overview of electricity demand forecasting techniques. Network and Complex Systems,”
vol. 3, no. 3, pp. 38-48, 2013.
[7] M. H.Amini, A. Kargarian, and O. Karabasoglu, “ARIMA-based decoupled time series forecasting of electric vehicle charging demand for stochastic power system operation.” Electric Power Systems Research, vol. 140, pp. 378-390, 2016.
[8] S. Voronin, and J. Partanen, “Forecasting electricity price and demand using a hybrid approach based on wavelet
transform,” ARIMA and neural networks. International Journal of Energy Research, vol. 38, no. 5, pp. 626-637, 2014.
[9] A. K. Topalli, I. Erkmen, and I. Topalli, “Intelligent short-term load forecasting in Turkey.” International Journal of Electrical Power & Energy Systems, vol. 28, no. 7, pp. 437-447, 2006.
[10] S. C. Bhattacharyya, and L. T. Thanh, “Shortterm electric load forecasting using an artificial neural network: case of Northern Vietnam.” International journal of energy research, vol. 28, no. 5, pp. 463-472, 2004.
[11] F. Cavallaro, “Electric load analysis using an artificial neural network.” International journal of energy research, vol. 29, no. 5, pp. 377-392, 2005.
[12] G. Nasr, E. Badr, and M. Younes, “Neural networks in forecasting electrical energy consumption: univariate and
multivariate approaches.” International Journal of Energy Research, vol. 26, no. 1, pp. 67-78, 2002.
[13] D. Singh, and S. Singh, “A self-selecting neural network for short-term load forecasting.” Electric Power Components and Systems, vol. 29, no. 2, pp. 117-130, 2001.
[14] A. Demiroren, and G. Ceylan, “Middle anatolian region short-term load forecasting using artificial neural networks.”
Electric Power Components and Systems, vol. 34, no. 6, pp. 707-724, 2006.
[15] T. Senjyu, H. Sakihara, Y. Tamaki, and K. Uezato, “Next-day load curve forecasting using neural network based on
similarity.” Electric Power Components and Systems, vol. 29, no. 10, pp. 939-948, 2001.
[16] A. Dedinec et al., “Deep belief network based electricity load forecasting: An analysis of Macedonian case.” Energy,
vol. 115, pp. 1688-1700, 2016.
[17] S. Fan, L. Chen, and W.-J. Lee, “Machine learning based switching model for electricity load forecasting.” Energy
Conversion and Management, vol. 49, no. 6, pp. 1331-1344, 2008.
[18] J. C. Sousa, H. M. Jorge, and L. P. Neves, “Shortterm load forecasting based on support vector regression and load
profiling.” International Journal of Energy Research, vol. 38, no. 3, pp. 350-362, 2014.
[19] M. Mohandes, “Support vector machines for shortterm electrical load forecasting.” International Journal of Energy
Research, vol. 26, no. 4, pp. 335-345, 2002.
[20] D. Srinivasan, and M. Lee, “Survey of hybrid fuzzy neural approaches to electric load forecasting. in Systems, Man and Cybernetics,” Intelligent Systems for the 21st Century, IEEE International Conference. 1995.
[21] S. Kazemi et al., “An evolutionary based adaptive neuro fuzzy inference system for intelligent shortterm load forecasting.” International Transactions in Operational Research, vol. 21, no. 2, pp. 311-326, 2014.
[22] K. -L. Ho et al., “Short term load forecasting of Taiwan power system using a knowledge-based expert system.” IEEE Transactions on Power Systems, vol. 5, no. 4, pp. 1214-1221, 1990.
[23] B. Kermanshahi, and H. Iwamiya, “Up to year 2020 load forecasting using neural nets.” International Journal of
Electrical Power & Energy Systems, vol. 24, no. 9, pp. 789-797, 2002.
[24] D. Bunn, and E. Farmer, “Review of short-term forecasting methods in the electric power industry.” Comparative models for electrical load forecasting, pp. 13-30, 1985.
[25] H. K. Alfares, and M. Nazeeruddin, “Electric load forecasting: literature survey and classification of methods.” International Journal of Systems Science, vol. 33, no. 1, pp. 23-34, 2002.
[26] I. Ben-Gal, Outlier detection, in Data mining and knowledge discovery handbook. Springer, 2005, pp. 131-146.
[27] Y. Yu et al., “Time series outlier detection based on sliding window prediction.” Mathematical Problems in Engineering, 2014.
[28] S. Xu et al., “An improved methodology for outlier detection in dynamic datasets.” AIChE Journal, vol. 61, no. 2, pp. 419-433, 2015.
[29] S. Hekimoglu, B. Erdogan, and R. Erenoglu, “A new outlier detection method considering outliers as model errors.”
Experimental Techniques, vol. 39, no. 1, pp. 57-68, 2015.
[30] C. Chen, and L. M. Liu, “Forecasting time series with outliers.” Journal of Forecasting, vol. 12, no. 1, pp. 13-35, 1993.
[31] H. Liu, S. Shah, and W. Jiang, “On-line outlier detection and data cleaning.” Computers & chemical engineering, vol. 28, no. 9, pp. 1635-1647, 2004.
[32] S. Trueck, R. Weron, and R. Wolff, Outlier treatment and robust approaches for modeling electricity spot prices, 2007.
[33] A. S. Hadi, “Identifying multiple outliers in multivariate data.” Journal of the Royal Statistical Society. Series B
(Methodological), pp. 761-771, 1992.
[34] A. S., Hadi, “A modification of a method for the detection of outliers in multivariate samples.” Journal of the Royal
Statistical Society. Series B (Methodological), pp. 393-396, 1994.
[35] J. H. Janssens, I. Flesch, and E.O. Postma. “Outlier detection with one-class classifiers from ML and KDD.” in Machine Learning and Applications, ICMLA’09. International Conference. 2009.
[36] G. J. Williams et al., “A Comparative Study of RNN for Outlier Detection in Data Mining.” in ICDM. 2002.
[37] S. Hawkins, et al., “Outlier detection using replicator neural networks.” in International Conference on Data Warehousing and Knowledge Discovery. Springer, 2002.
[38] J. Chen et al., “Automated load curve data cleansing in power systems.” IEEE Transactions on Smart Grid, vol. 1,
no. 2, pp. 213-221, 2010.
[39] A. Tidemann, and H. Langseth, “Effects of data cleansing on load prediction algorithms.” in Computational Intelligence Applications In Smart Grid (CIASG), IEEE Symposium. 2013.
[40] G. Tang et al., “From landscape to portrait: a new approach for outlier detection in load curve data.” IEEE Transactions on Smart Grid, vol. 5, no. 4, pp. 1764-1773, 2014.
[41] R. M. do Nascimento et al., “Outliers’ detection and filling algorithms for smart metering centers.” in Transmission and Distribution Conference and Exposition (T&D), IEEE PES. 2012.
[42] G. Mateos, and G. B. Giannakis, “Load curve data cleansing and imputation via sparsity and low rank.” IEEE Transactions on Smart Grid, vol. 4, no. 4, pp. 2347-2355, 2013.
[43] V. J. Hodge, and J. Austin, “A survey of outlier detection methodologies.” Artificial intelligence review, vol. 22, no. 2, pp. 85-126, 2004.
[44] R. Rustum, and A. J. Adeloye, “Replacing outliers and missing values from activated sludge data using Kohonen
self-organizing map.” Journal of Environmental Engineering, vol. 133, no. 9, pp. 909-916, 2007.
[45] H. R. Maier, and G. C. Dandy, “The use of artificial neural networks for the prediction of water quality parameters.” Water Resour Res, vol. 32, no. 4, pp. 1013-1022, 1996.
[46] A. Adeloye, and A. De Munari, “Artificial neural network based generalized storage–yield–reliability models using the Levenberg–Marquardt algorithm.” Journal of Hydrology, vol. 326, no. 1, pp. 215-230, 2006.
[47] I. L. MacDonald, and W. Zucchini, Hidden Markov and other models for discrete-valued time series, vol. 110. CRC
Press, 1997.
[48] A. C. Harvey, Forecasting, structural time series models and the Kalman filter. Cambridge university press, 1990.
[49] Z. Zhang et al., “A study on the method for cleaning and repairing the probe vehicle data.” IEEE Transactions on
Intelligent Transportation Systems, vol. 14, no. 1, pp. 419-427, 2013.
[50] M. Batmend, and D. Perdukova, “Linear regression based real-time filtering.” Advances in Electrical and Electronic
Engineering, vol. 11, no. 6, pp. 487, 2013.
[51] J. Connor, “A robust neural network filter for electricity demand prediction.” Journal of Forecasting, vol. 15, no. 6,
pp. 437-458, 1996.
[52] R. Weron, “Modeling and Forecasting Electricity Loads.” Modeling and Forecasting Electricity Loads and Prices:
A Statistical Approach, pp. 67-100, 2006.
[53] W. S. Cleveland, S. J. Devlin, and E. Grosse, “Regression by local fitting: methods, properties, and computational
algorithms.” Journal of econometrics, vol. 37, no. 1, pp. 87-114, 1988.
[54] W. S. Cleveland, and S. J. Devlin, “Locally weighted regression: an approach to regression analysis by local fitting.”
Journal of the American statistical association, vol. 83, no. 403, pp. 596-610, 1988.
[55] W. S. Cleveland, “Robust locally weighted regression and smoothing scatterplots.” Journal of the American statistical association, vol. 74, no. 368, pp. 829-836, 1979.
[2] R. Sadownik, and E. P. Barbosa, “Shortterm forecasting of industrial electricity consumption in Brazil,” Journal of
Forecasting, vol. 18, no. 3, pp. 215-224, 1999.
[3] Y. Ohtsuka, and K. Kakamu, “SpaceTime Model versus VAR Model: Forecasting Electricity demand in Japan.” Journal
of Forecasting, vol. 32, no. 1, pp. 75-85, 2013.
[4] U. D. Caprio et al., “Short term load forecasting in electric power systems: a comparison of ARMA models and extended wiener filtering.” Journal of Forecasting, vol. 2, no. 1, pp. 59-76, 1983.
[5] G. J. Janacek, and L. Swift, Time series: forecasting, simulation, applications. Ellis horwood, 1993.
[6] A. K. Singh et al., “An overview of electricity demand forecasting techniques. Network and Complex Systems,”
vol. 3, no. 3, pp. 38-48, 2013.
[7] M. H.Amini, A. Kargarian, and O. Karabasoglu, “ARIMA-based decoupled time series forecasting of electric vehicle charging demand for stochastic power system operation.” Electric Power Systems Research, vol. 140, pp. 378-390, 2016.
[8] S. Voronin, and J. Partanen, “Forecasting electricity price and demand using a hybrid approach based on wavelet
transform,” ARIMA and neural networks. International Journal of Energy Research, vol. 38, no. 5, pp. 626-637, 2014.
[9] A. K. Topalli, I. Erkmen, and I. Topalli, “Intelligent short-term load forecasting in Turkey.” International Journal of Electrical Power & Energy Systems, vol. 28, no. 7, pp. 437-447, 2006.
[10] S. C. Bhattacharyya, and L. T. Thanh, “Shortterm electric load forecasting using an artificial neural network: case of Northern Vietnam.” International journal of energy research, vol. 28, no. 5, pp. 463-472, 2004.
[11] F. Cavallaro, “Electric load analysis using an artificial neural network.” International journal of energy research, vol. 29, no. 5, pp. 377-392, 2005.
[12] G. Nasr, E. Badr, and M. Younes, “Neural networks in forecasting electrical energy consumption: univariate and
multivariate approaches.” International Journal of Energy Research, vol. 26, no. 1, pp. 67-78, 2002.
[13] D. Singh, and S. Singh, “A self-selecting neural network for short-term load forecasting.” Electric Power Components and Systems, vol. 29, no. 2, pp. 117-130, 2001.
[14] A. Demiroren, and G. Ceylan, “Middle anatolian region short-term load forecasting using artificial neural networks.”
Electric Power Components and Systems, vol. 34, no. 6, pp. 707-724, 2006.
[15] T. Senjyu, H. Sakihara, Y. Tamaki, and K. Uezato, “Next-day load curve forecasting using neural network based on
similarity.” Electric Power Components and Systems, vol. 29, no. 10, pp. 939-948, 2001.
[16] A. Dedinec et al., “Deep belief network based electricity load forecasting: An analysis of Macedonian case.” Energy,
vol. 115, pp. 1688-1700, 2016.
[17] S. Fan, L. Chen, and W.-J. Lee, “Machine learning based switching model for electricity load forecasting.” Energy
Conversion and Management, vol. 49, no. 6, pp. 1331-1344, 2008.
[18] J. C. Sousa, H. M. Jorge, and L. P. Neves, “Shortterm load forecasting based on support vector regression and load
profiling.” International Journal of Energy Research, vol. 38, no. 3, pp. 350-362, 2014.
[19] M. Mohandes, “Support vector machines for shortterm electrical load forecasting.” International Journal of Energy
Research, vol. 26, no. 4, pp. 335-345, 2002.
[20] D. Srinivasan, and M. Lee, “Survey of hybrid fuzzy neural approaches to electric load forecasting. in Systems, Man and Cybernetics,” Intelligent Systems for the 21st Century, IEEE International Conference. 1995.
[21] S. Kazemi et al., “An evolutionary based adaptive neuro fuzzy inference system for intelligent shortterm load forecasting.” International Transactions in Operational Research, vol. 21, no. 2, pp. 311-326, 2014.
[22] K. -L. Ho et al., “Short term load forecasting of Taiwan power system using a knowledge-based expert system.” IEEE Transactions on Power Systems, vol. 5, no. 4, pp. 1214-1221, 1990.
[23] B. Kermanshahi, and H. Iwamiya, “Up to year 2020 load forecasting using neural nets.” International Journal of
Electrical Power & Energy Systems, vol. 24, no. 9, pp. 789-797, 2002.
[24] D. Bunn, and E. Farmer, “Review of short-term forecasting methods in the electric power industry.” Comparative models for electrical load forecasting, pp. 13-30, 1985.
[25] H. K. Alfares, and M. Nazeeruddin, “Electric load forecasting: literature survey and classification of methods.” International Journal of Systems Science, vol. 33, no. 1, pp. 23-34, 2002.
[26] I. Ben-Gal, Outlier detection, in Data mining and knowledge discovery handbook. Springer, 2005, pp. 131-146.
[27] Y. Yu et al., “Time series outlier detection based on sliding window prediction.” Mathematical Problems in Engineering, 2014.
[28] S. Xu et al., “An improved methodology for outlier detection in dynamic datasets.” AIChE Journal, vol. 61, no. 2, pp. 419-433, 2015.
[29] S. Hekimoglu, B. Erdogan, and R. Erenoglu, “A new outlier detection method considering outliers as model errors.”
Experimental Techniques, vol. 39, no. 1, pp. 57-68, 2015.
[30] C. Chen, and L. M. Liu, “Forecasting time series with outliers.” Journal of Forecasting, vol. 12, no. 1, pp. 13-35, 1993.
[31] H. Liu, S. Shah, and W. Jiang, “On-line outlier detection and data cleaning.” Computers & chemical engineering, vol. 28, no. 9, pp. 1635-1647, 2004.
[32] S. Trueck, R. Weron, and R. Wolff, Outlier treatment and robust approaches for modeling electricity spot prices, 2007.
[33] A. S. Hadi, “Identifying multiple outliers in multivariate data.” Journal of the Royal Statistical Society. Series B
(Methodological), pp. 761-771, 1992.
[34] A. S., Hadi, “A modification of a method for the detection of outliers in multivariate samples.” Journal of the Royal
Statistical Society. Series B (Methodological), pp. 393-396, 1994.
[35] J. H. Janssens, I. Flesch, and E.O. Postma. “Outlier detection with one-class classifiers from ML and KDD.” in Machine Learning and Applications, ICMLA’09. International Conference. 2009.
[36] G. J. Williams et al., “A Comparative Study of RNN for Outlier Detection in Data Mining.” in ICDM. 2002.
[37] S. Hawkins, et al., “Outlier detection using replicator neural networks.” in International Conference on Data Warehousing and Knowledge Discovery. Springer, 2002.
[38] J. Chen et al., “Automated load curve data cleansing in power systems.” IEEE Transactions on Smart Grid, vol. 1,
no. 2, pp. 213-221, 2010.
[39] A. Tidemann, and H. Langseth, “Effects of data cleansing on load prediction algorithms.” in Computational Intelligence Applications In Smart Grid (CIASG), IEEE Symposium. 2013.
[40] G. Tang et al., “From landscape to portrait: a new approach for outlier detection in load curve data.” IEEE Transactions on Smart Grid, vol. 5, no. 4, pp. 1764-1773, 2014.
[41] R. M. do Nascimento et al., “Outliers’ detection and filling algorithms for smart metering centers.” in Transmission and Distribution Conference and Exposition (T&D), IEEE PES. 2012.
[42] G. Mateos, and G. B. Giannakis, “Load curve data cleansing and imputation via sparsity and low rank.” IEEE Transactions on Smart Grid, vol. 4, no. 4, pp. 2347-2355, 2013.
[43] V. J. Hodge, and J. Austin, “A survey of outlier detection methodologies.” Artificial intelligence review, vol. 22, no. 2, pp. 85-126, 2004.
[44] R. Rustum, and A. J. Adeloye, “Replacing outliers and missing values from activated sludge data using Kohonen
self-organizing map.” Journal of Environmental Engineering, vol. 133, no. 9, pp. 909-916, 2007.
[45] H. R. Maier, and G. C. Dandy, “The use of artificial neural networks for the prediction of water quality parameters.” Water Resour Res, vol. 32, no. 4, pp. 1013-1022, 1996.
[46] A. Adeloye, and A. De Munari, “Artificial neural network based generalized storage–yield–reliability models using the Levenberg–Marquardt algorithm.” Journal of Hydrology, vol. 326, no. 1, pp. 215-230, 2006.
[47] I. L. MacDonald, and W. Zucchini, Hidden Markov and other models for discrete-valued time series, vol. 110. CRC
Press, 1997.
[48] A. C. Harvey, Forecasting, structural time series models and the Kalman filter. Cambridge university press, 1990.
[49] Z. Zhang et al., “A study on the method for cleaning and repairing the probe vehicle data.” IEEE Transactions on
Intelligent Transportation Systems, vol. 14, no. 1, pp. 419-427, 2013.
[50] M. Batmend, and D. Perdukova, “Linear regression based real-time filtering.” Advances in Electrical and Electronic
Engineering, vol. 11, no. 6, pp. 487, 2013.
[51] J. Connor, “A robust neural network filter for electricity demand prediction.” Journal of Forecasting, vol. 15, no. 6,
pp. 437-458, 1996.
[52] R. Weron, “Modeling and Forecasting Electricity Loads.” Modeling and Forecasting Electricity Loads and Prices:
A Statistical Approach, pp. 67-100, 2006.
[53] W. S. Cleveland, S. J. Devlin, and E. Grosse, “Regression by local fitting: methods, properties, and computational
algorithms.” Journal of econometrics, vol. 37, no. 1, pp. 87-114, 1988.
[54] W. S. Cleveland, and S. J. Devlin, “Locally weighted regression: an approach to regression analysis by local fitting.”
Journal of the American statistical association, vol. 83, no. 403, pp. 596-610, 1988.
[55] W. S. Cleveland, “Robust locally weighted regression and smoothing scatterplots.” Journal of the American statistical association, vol. 74, no. 368, pp. 829-836, 1979.