Journal of Korea Society of Industrial Information Systems (한국산업정보학회논문지)

Korea Society of Industrial Information Systems (한국산업정보학회)
권호 표지
DOI QR코드

DOI QR Code

Large Language Models-based Feature Extraction for Short-Term Load Forecasting

거대언어모델 기반 특징 추출을 이용한 단기 전력 수요량 예측 기법

  • Jaeseung Lee ;
  • Jehyeok Rew
  • 이재승 (고려대학교 전기전자공학과) ;
  • 유제혁 (덕성여자대학교 데이터사이언스학과)
  • Received : 2024.05.18
  • Accepted : 2024.06.14
  • Published : 2024.06.30
Download PDF
⟨ Previous Next ⟩

Abstract

Accurate electrical load forecasting is important to the effective operation of power systems in smart grids. With the recent development in machine learning, artificial intelligence-based models for predicting power demand are being actively researched. However, since existing models get input variables as numerical features, the accuracy of the forecasting model may decrease because they do not reflect the semantic relationship between these features. In this paper, we propose a scheme for short-term load forecasting by using features extracted through the large language models for input data. We firstly convert input variables into a sentence-like prompt format. Then, we use the large language model with frozen weights to derive the embedding vectors that represent the features of the prompt. These vectors are used to train the forecasting model. Experimental results show that the proposed scheme outperformed models based on numerical data, and by visualizing the attention weights in the large language models on the prompts, we identified the information that significantly influences predictions.

스마트 그리드에서 전력 시스템을 효과적으로 운영하기 위해서는 전력 수요량을 정확히 예측하는 것이 중요하다. 최근 기계학습 기술의 발달로, 인공지능 기반의 전력 수요량 예측 모델이 활발히 연구되고 있다. 하지만, 기존 모델들은 모든 입력변수를 수치화하여 입력하기 때문에, 이러한 수치들 사이의 의미론적 관계를 반영하지 못해 예측 모델의 정확도가 하락할 수 있다. 본 논문은 입력 데이터에 대하여 거대언어모델을 통해 추출한 특징을 이용하여 단기 전력 수요량을 예측하는 기법을 제안한다. 먼저, 입력변수를 문장 형식의 프롬프트로 변환한다. 이후, 가중치가 동결된 거대언어모델을 이용하여 프롬프트에 대한 특징을 나타내는 임베딩 벡터를 도출하고, 이를 입력으로 받은 모델을 학습하여 예측을 수행한다. 실험 결과, 제안 기법은 수치형 데이터에 기반한 예측 모델에 비해 높은 성능을 보였고, 프롬프트에 대한 거대언어모델의 주의집중 가중치를 시각화함으로써 예측에 있어 주요한 영향을 미친 정보를 확인하였다.

Keywords

References

  1. Aisyah, S., Simaremare, A., Adytia, D., Aditya, I. and Alamsyah, A. (2022). Exploratory Weather Data Analysis for Electricity Load Forecasting Using SVM and GRNN, Case Study in Bali, Indonesia. Energies, 15(10), https://doi.org/10.3390/en15103566
  2. Chen, T. and Guestrin, C. (2016). XGBoost: A Scalable Tree Boosting System. ACM SIGKDD Intenrational Conference on Knowledge Discovery and Data Mining, Aug. 13 - 17, San Francisco, CA, USA, pp. 785-794.
  3. Chodakowska, E., Nazarko, J. and Nazarko, L. (2021). ARIMA Models in Electrical Load Forecasting and Their Robustness to Noise. Energies, 14(23), https://doi.org/10.3390/en14237952
  4. Clark, K., Luong, M., Le, Q. and Manning, C. (2020). ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators. International Conference on Learning Representations, Apr. 26-30, Virtual, pp. 1-18.
  5. Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzman, F., Grave, E., Ott, M., Zettlemoyer, L. and Stoyanov, V. (2020). Unsupervised Cross-lingual Representation Learning at Scale. Annual Meeting of the Association for Computational Linguistics, Jul. 06 - 08, Online, pp. 8440-8451.
  6. Dehalwar, V., Kalam, A., Kolhe, M. and Zayegh, A. (2016). Electricity Load Forecasting for Urban Area Using Weather Forecast Information. IEEE International Conference on Power and Renewable Energy, Oct. 21 - 23, Shanghai, China, pp. 355-359.
  7. Devlin, J., Chang, M., Lee, K. and Toutanova, K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Conference of the North American Chapter of the Association for Computational Linguistics, Minneapolis, MN, USA, pp. 4171-4186.
  8. Dudek, G. (2022). A Comprehensive Study of Random Forest for Short-Term Load Forecasting. Energies, 15(20), https://doi.org/10.3390/en15207547
  9. Hong, T., Pinson, P. and Fan, S. (2014). Global Energy Forecasting Competition 2012. International Journal of Forecasting, 30(2), 357-363, https://doi.org/10.1016/j.ijforecast.2013.07.001
  10. Ke, G., Meng, Q., Finley, T., Wang, T., Chen, W., Ma, W., Ye, Q. and Liu, T. (2017). LightGBM: A Highly Efficient Gradient Boosting Decision Tree. International Conference on Neural Information Processing Systems, Dec. 04 - 09, Long Beach, CA, USA, pp. 3149-3157.
  11. Kim, H. and Yu, Y. (2023). Development of a Regulatory Q&A System for KAERI UtilizingDocument Search Algorithms and Large Language Model. Journal of Korea Society of Industrial Information Systems, 28(5), 31-39, http://doi.org/10.9723/jksiis.2023.28.5.031
  12. Kim, H., Jang, J., Kim, J. and Kim, K. (2023). Predicting Forest Fires Using Machine Learning Considering Human Factors. Journal of Korea Society of Industrial Information Systems, 28(5), 109-126, https://doi.org/10.9723/jksiis.2023.28. 5.109
  13. Kumar, M. and Pal, N. (2023). Machine Learning-based Electric Load Forecasting for Peak Demand Control in Smart Grid. Computers, Materials & Continua, 74(3), https://doi.org/10.32604/cmc.2022.032971
  14. Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P. and Soricut, R. (2020). ALBERT: A Lite BERT for Self-supervised Learning of Language Representations. International Conference on Learning Representations, Apr. 26 - 30, Virtual, pp. 1-17.
  15. Lee, C. and Ko, C. (2011). Short-term Load Forecasting Using Lifting Scheme and ARIMA Models. Expert Systems with Applications, 38(5), 5902-5911, https://doi.org/10.1016/j.eswa.2010.11.033
  16. Lee, D. (2023). The Prediction of Survival of Breast Cancer Patients Based on Machine Learning Using Health Insurance Claim Data. Journal of Korea Society of Industrial Information Systems, 28(2), 1-9, https://doi.org/10.9723/jksiis.2023.28.2.001
  17. Lee, H. and Shin, Y. (2013). Forecasting Electric Power Demand Using Census Information and Electric Power Load. Journal of Korea Society of Industrial Information Systems, 18(3), 35-46, https://doi.org/10.9723/jksiis.2013.18.3.035
  18. Lee, J., Moon, J., Park, S. and Hwang, E. (2022). Photovoltaic Power Forecasting Scheme Based on CTGAN Oversampling Considering Weather Data Imbalance Problem. Korea Software Congress, Dec. 20 - 23, Jeju, Republic of Korea, pp. 1379-1381.
  19. Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L. and Stoyanov, V. (2019). RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv preprint, arXiv:1907.11692
  20. Louppe, G. (2014). Understanding Random Forests: From Theory to Practice. University of Liege, Belgium, Ph.D. Dissertation.
  21. Moon, J., Jung, S., Park, S. and Hwang, E. (2020a). Conditional Tabular GAN-Based Two-Stage Data Generation Scheme for Short-Term Load Forecasting. IEEE Access, 8, 205327-205339, https://doi.org/10.1109/ACCESS.2020.3037063
  22. Moon, J., Kim, J., Kang, P. and Hwang, E. (2020b). Solving the Cold-Start Problem in Short-Term Load Forecasting Using Tree-Based Methods. Energies, 13(4), https://doi.org/10.3390/en13040886
  23. Oh, J., Ham, D., Lee, Y. and Kim, G. (2019). Short-term Load Forecasting Using XGBoost and the Analysis of Hyperparameters. The Transactions of the Korean Institute of Electrical Engineers, 68(9), 1073-1078, https://doi.org/10.5370/KIEE.2019.68.9.1073
  24. Raju, M. and Laxmi, A. (2020). IOT Based Online Load Forecasting Using Machine Learning Algorithms. Procedia Computer Science, 171, 551-560, https://doi.org/10.1016/j.procs.2020.04.059
  25. Saglam, M., Lv, X., Spataru, C. and Karama, O. (2024). Instantaneous Electricity Peak Load Forecasting Using Optimization and Machine Learning. Energies, 17(4), https://doi.org/10.3390/en17040777
  26. Sanh, V., Debut, L., Chaumond, J. and Wolf, T. (2019). DistilBERT, A Distilled Version of BERT: Smaller, Faster, Cheaper and Lighter. arXiv preprint, arXiv:1910.01108.
  27. Son, S. (2023). The Evolution and Future Role of Smart Grids. Fall Conference on Smart Grid Research, Oct. 20, Seoul, Republic of Korea, pp. 1-12.
  28. Tan, Y., Teng, Z., Zhang, C., Zuo, G., Wang, Z. and Zhao, Z. (2021). Long-Term Load Forecasting Based on Feature Fusion and LightGBM. IEEE International Conference on Power and Energy Applications, Oct. 09-11, Busan, Republic of Korea, pp. 104-109.
  29. Wang, W. (2016). Improved Short Term Load Forecasting of Power System Based on ARMA model. International Conference on Engineering Management, Nov. 26 - 27, Guangzhou, China, pp. 12-19.
  30. Zhu, Y., Yuan, H., Wang, S., Liu, J., Liu, W., Deng, C., Chen, H., Dou, Z. and Wen, J. (2023). Large Language Models for Information Retrieval: A Survey. arXiv preprint, arXiv:2308.07107