Browse > Article
http://dx.doi.org/10.20465/KIOTS.2022.8.5.017

HTML Tag Depth Embedding: An Input Embedding Method of the BERT Model for Improving Web Document Reading Comprehension Performance  

Mok, Jin-Wang (Division of Computer Engineering, Baekseok University)
Jang, Hyun Jae (Division of Computer Engineering, Baekseok University)
Lee, Hyun-Seob (Division of Advanced IT, Baekseok University)
Publication Information
Journal of Internet of Things and Convergence / v.8, no.5, 2022 , pp. 17-25 More about this Journal
Abstract
Recently the massive amount of data has been generated because of the number of edge devices increases. And especially, the number of raw unstructured HTML documents has been increased. Therefore, MRC(Machine Reading Comprehension) in which a natural language processing model finds the important information within an HTML document is becoming more important. In this paper, we propose HTDE(HTML Tag Depth Embedding Method), which allows the BERT to train the depth of the HTML document structure. HTDE makes a tag stack from the HTML document for each input token in the BERT and then extracts the depth information. After that, we add a HTML embedding layer that takes the depth of the token as input to the step of input embedding of BERT. Since tokenization using HTDE identifies the HTML document structures through the relationship of surrounding tokens, HTDE improves the accuracy of BERT for HTML documents. Finally, we demonstrated that the proposed idea showing the higher accuracy compared than the accuracy using the conventional embedding of BERT.
Keywords
Natural Language Processing; Machine Reading Comprehension; Embeddings; HTML; BERT;
Citations & Related Records
Times Cited By KSCI : 1  (Citation Analysis)
연도 인용수 순위
1 J.Lee, "Analysis of the Hardware Structures of the IoT Device Platforms for the Minimal Power Consumption" Journal of Internet of Things and Convergence, Vol.6, No.2, pp.11-18. 2020.
2 S.H.Lee and D.W.Lee, "A Software Engineering-Based Software Development Progress Analysis in IoT Environment" Journal of Internet of Things and Convergence, Vol.6, No.2, pp.87-92. 2020.   DOI
3 T.Kwiatkowski, J.Palomaki, O.Redfield, M.Collins, A.Parikh, C.Alberti, D.Epstein, I.Polosukhin, J.Devlin, K.Lee, K.Toutanova, L.Jones, M.Kelcey, M.W.Chang, A.M.Dai, J.Uszkoreit, Q.Le and S.Petrov, "Natural questions: a benchmark for question answering research." Transactions of the Association for Computational Linguistics, Vol.7, pp.453-466. 2019.   DOI
4 E.Choi, H.He, M.Iyyer, M.Yatskar, W.Yih, Y.Choi, P.Liang and L.Zettlemoyer, "QuAC: Question answering in context." Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp.2174-2184, 2018.
5 P.Rajpurkar, J.Zhang, K.Lopyrev, and P.Liang, "Squad: 100,000+ questions for machine comprehension of text." Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp.2383-2392, 2016.
6 S.Y.Lim, M.J.Kim, and J.Y.Lee. "Korquad: Korean qa dataset for machine comprehension." Proceeding of the Conference of the Korea Information Science Society, Vol.45, No.2, pp.539-541, 2018.
7 Y.M.Kim, S.Y.Lim, H.J.Lee, S.Y.Park and M.J.Kim, "KorQuAD 2.0: Korean QA dataset for web document machine comprehension." Proceeding of the Conference of the Korea Information Science Society, Vol.47, No.06, pp.577-586, 2020.
8 J.Devlin, M.W.Chang, K.Lee and K.Toutanova, "Bert: Pre-training of deep bidirectional transformers for language understanding." Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Vol.1, pp.4171-4186, 2019.
9 A.Asai, X.Yu, J.Kasai and H.Hajishirzi, "One question answering model for many languages with cross-lingual dense passage retrieval." Advances in Neural Information Processing Systems, Vol.34, pp.7547-7560, 2021.
10 D.Chen, A.Fisch, J.Weston and A.Bordes, "Reading wikipedia to answer open-domain questions." Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, Vol.1, pp.1870-1879. 2017.
11 H.S.Lee, "A Prediction-Based Data Read Ahead Policy using Decision Tree for improving the performance of NAND flash memory based storage devices" Journal of Internet of Things and Convergence, Vol.8, No.4, pp.9-15. 2022.   DOI
12 Z.Yang, P.Qi, S.Zhang, Y.Bengio, W.W.Cohen, R.Salakhutdinov and C.D.Manning, "HotpotQA: A dataset for diverse, explainable multi-hop question answering." Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp.2369-2380, 2018.
13 I.Sutskever, O.Vinyals, and Q.V.Le, "Sequence to sequence learning with neural networks." In Proceedings of the 27th International Conference on Neural Information Processing Systems, Vol.2, pp.3104-3112, 2014.
14 D.Bahdanau, K.H.Cho and Y.Bengio, "Neural machine translation by jointly learning to align and translate.", 3rd International Conference on Learning Representations, 2015.
15 A.Vaswani, N.Shazeer, N.Parmar, J.Uszkoreit, L.Jones, A.N.Gomez, L.Kaiser and I.Polosukhin, "Attention is all you need." Advances in neural information processing systems, Vol.30, pp.5998-6008, 2017.
16 B.Wang, L.Shang, C.Lioma, X.Jiang, H.Yang, Q.Liu, and J.G.Simonsen, "On position embeddings in bert." International Conference on Learning Representations, 2020.