• The Journal of Korean Institute of Next Generation Computing
• /
• v.13 no.2
• /
• pp.18-25
• /
• 2017

In computer science, an encoding is a standardization of converting information to one format for audio, video or text. Therefore, the encoding information of the data should be known to open and read it and there are algorithms detecting encoder of the data. However, some informations of data could be disappeared by packet loss when transmitted on network, especially, if the data is snatched by packet sniffing or eavesdropping from wireless communications. In this paper, we improve the performance of encoding detection algorithm of 'uchardet' program for multi-byte encoded data with errors based on bit-shift algorithm. To simulate the performance, we generated Korean and Japanese text data with errors that is removed some random bits at random positions. Then the detection algorithm are tested using the data and 'uchardet-bitshift' showed better performance than 'uchardet'. When Korean texts are used, 'uchardet' could detect perfectly with ≤0.005% errors but it showed 0% detection rate with ≥1% errors while 'uchardet-bitshift' detected perfectly with ≤0.05% errors and it showed correct detection cases with ≥1% errors. Japanese texts with errors tend to report falsely as Chinese encoding because Japanese texts include lots of Chinese characters. As a results, we improved encoding detection algorithms by applying bit shift operation.

• The Journal of Korean Institute of Next Generation Computing
• /
• v.13 no.5
• /
• pp.124-130
• /
• 2017

Character encoding is the method used to represent characters or symbols on a computer, and there are many encoding detection software tools. For the widely used encoding detection software"uchardet", the accuracy of encoding detection of unmodified normal text document is 91.39%, but the accuracy of language detection is only 32.09%. Also, if a text document is encrypted by substitution, the accuracy of encoding detection is 3.55% and the accuracy of language detection is 0.06%. Therefore, in this paper, we propose encoding and language detection of text document using the deep learning algorithm called LSTM(Long Short-Term Memory). The results of LSTM are better than encoding detection software"uchardet". The accuracy of encoding detection of normal text document using the LSTM is 99.89% and the accuracy of language detection is 99.92%. Also, if a text document is encrypted by substitution, the accuracy of encoding detection is 99.26%, the accuracy of language detection is 99.77%.