Browse > Article
http://dx.doi.org/10.3745/KTSDE.2021.10.12.579

Style-Based Transformer for Time Series Forecasting  

Kim, Dong-Keon (성균관대학교 소프트웨어학과)
Kim, Kwangsu (성균관대학교 소프트웨어학과)
Publication Information
KIPS Transactions on Software and Data Engineering / v.10, no.12, 2021 , pp. 579-586 More about this Journal
Abstract
Time series forecasting refers to predicting future time information based on past time information. Accurately predicting future information is crucial because it is used for establishing strategies or making policy decisions in various fields. Recently, a transformer model has been mainly studied for a time series prediction model. However, the existing transformer model has a limitation in that it has an auto-regressive structure in which the output result is input again when the prediction sequence is output. This limitation causes a problem in that accuracy is lowered when predicting a distant time point. This paper proposes a sequential decoding model focusing on the style transformation technique to handle these problems and make more precise time series forecasting. The proposed model has a structure in which the contents of past data are extracted from the transformer-encoder and reflected in the style-based decoder to generate the predictive sequence. Unlike the decoder structure of the conventional auto-regressive transformer, this structure has the advantage of being able to more accurately predict information from a distant view because the prediction sequence is output all at once. As a result of conducting a prediction experiment with various time series datasets with different data characteristics, it was shown that the model presented in this paper has better prediction accuracy than other existing time series prediction models.
Keywords
Time Series Forecasting; Transformer; Generative Decoder; Style Transfer;
Citations & Related Records
연도 인용수 순위
  • Reference
1 D. P. Kingma and J. Ba, "Adam: A method for stochastic optimization," arXiv preprint arXiv:1412.6980, 2014.
2 A. B. Geva, "ScaleNet-multiscale neural-network architecture for time series prediction," IEEE Transactions on Neural Networks, Vol.9, No.6, pp.1471-1482, 1998.   DOI
3 A. Tokgoz and G. unal, "A RNN based time series approach for forecasting turkish electricity load," 2018 26th Signal Processing and Communications Applications Conference (SIU), IEEE, 2018.
4 I. Koprinska, D. Wu, and Z. Wang, "Convolutional neural networks for energy time series forecasting," 2018 International Joint Conference on Neural Networks (IJCNN), IEEE, 2018.
5 J. Pamina and B. Raja, "Survey on deep learning algorithms," International Journal of Emerging Technology and Innovative Engineering, Vol.5, No.1, 2019.
6 Y. Lin, I. Koprinska, and M. Rana, "SpringNet: Transformer and spring DTW for time series forecasting," International Conference on Neural Information Processing. Springer, Cham, 2020.
7 N. Chen, S. Watanabe, J. Villalba, P. Zelasko, and N. Dehak, "Non-autoregressive transformer for speech recognition," IEEE Signal Processing Letters, Vol.28, pp.121-125, 2020.
8 I. Goodfellow, et al., "Generative adversarial nets," Advances in Neural Information Processing Systems, Vol.27, 2014.
9 G. Lai, W.-C. Chang, Y. Yang, and H. Liu, "Modeling long-and short-term temporal patterns with deep neural networks," The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, 2018.
10 C.-J. Huang and P.-H. Kuo, "A deep cnn-lstm model for particulate matter (PM2. 5) forecasting in smart cities," Sensors, Vol.18, No.7, pp.2220, 2018.   DOI
11 Y. G. Cinar, H. Mirisaee, P. Goswami, E. Gaussier, A. Ait-Bachir, and V. Strijov, "Position-based content attention for time series forecasting with sequence-to-sequence rnns," International Conference on neural Information Processing, Springer, Cham, 2017.
12 J. L. Ba, J. R. Kiros, and G. E. Hinton, "Layer normalization," arXiv preprint arXiv:1607.06450, 2016.
13 T. Karras, S. Laine, and T. Aila, "A style-based generator architecture for generative adversarial networks," Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019.
14 J. J. Dabrowski, Y. Zhang, and A. Rahman, "ForecastNet: A time-variant deep feed-forward neural network architecture for multi-step-ahead time-series forecasting," International Conference on Neural Information Processing, Springer, Cham, 2020.
15 A. Vaswani, et al., "Attention is all you need," Advances in Neural Information Processing Systems, 2017.
16 S. Li, et al., "Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting," Advances in Neural Information Processing Systems, Vol.32, pp.5243-5253, 2019.
17 L. Cai, K. Janowicz, G. Mai, B. Yan, and R. Zhu, "Traffic transformer: Capturing the continuity and periodicity of time series for traffic forecasting," Transactions in GIS, Vol.24., No.3, pp.736-755, 2020.   DOI
18 F. Giuliari, I. Hasan, M. Cristani, and F. Galasso, "Transformer networks for trajectory forecasting," 2020 25th International Conference on Pattern Recognition (ICPR). IEEE, 2021.
19 B. L. Bowerman and R. T. O'connell, "Time series and forecasting," North Scituate, MA: Duxbury Press, 1979.
20 J. Contreras, R. Espinola, F. J. Nogales, and A. J. Conejo, "ARIMA models to predict next-day electricity prices," IEEE Transactions on Power Systems, Vol.18, No.3, pp.1014-1020, 2003.   DOI
21 A. I. McLeod, "Diagnostic checking of periodic autoregression models with application," Journal of Time Series Analysis, Vol.15, No.2, pp.221-233, 1994.   DOI
22 R. B. Darlington and A. F. Hayes, "Regression analysis and linear models," New York, NY: Guilford, 2017.
23 L. A. Gatys, A. S. Ecker, and M. Bethge, "Image style transfer using convolutional neural networks," Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016.
24 J. T. Connor, R. D. Martin, and L. E. Atlas, "Recurrent neural networks and robust time series prediction," IEEE Transactions on Neural Networks, Vol.5, No.2, pp.240-254, 1994.   DOI
25 R. J. Frank, N. Davey, and S. P. Hunt, "Time series prediction and neural networks," Journal of Intelligent and Robotic Systems, Vol.31, No.1, pp.91-103, 2001.   DOI