Video Analysis System for Action and Emotion Detection by Object with Hierarchical Clustering based Re-ID (계층적 군집화 기반 Re-ID를 활용한 객체별 행동 및 표정 검출용 영상 분석 시스템)
-
- Journal of Intelligence and Information Systems
- /
- v.28 no.1
- /
- pp.89-106
- /
- 2022
Recently, the amount of video data collected from smartphones, CCTVs, black boxes, and high-definition cameras has increased rapidly. According to the increasing video data, the requirements for analysis and utilization are increasing. Due to the lack of skilled manpower to analyze videos in many industries, machine learning and artificial intelligence are actively used to assist manpower. In this situation, the demand for various computer vision technologies such as object detection and tracking, action detection, emotion detection, and Re-ID also increased rapidly. However, the object detection and tracking technology has many difficulties that degrade performance, such as re-appearance after the object's departure from the video recording location, and occlusion. Accordingly, action and emotion detection models based on object detection and tracking models also have difficulties in extracting data for each object. In addition, deep learning architectures consist of various models suffer from performance degradation due to bottlenects and lack of optimization. In this study, we propose an video analysis system consists of YOLOv5 based DeepSORT object tracking model, SlowFast based action recognition model, Torchreid based Re-ID model, and AWS Rekognition which is emotion recognition service. Proposed model uses single-linkage hierarchical clustering based Re-ID and some processing method which maximize hardware throughput. It has higher accuracy than the performance of the re-identification model using simple metrics, near real-time processing performance, and prevents tracking failure due to object departure and re-emergence, occlusion, etc. By continuously linking the action and facial emotion detection results of each object to the same object, it is possible to efficiently analyze videos. The re-identification model extracts a feature vector from the bounding box of object image detected by the object tracking model for each frame, and applies the single-linkage hierarchical clustering from the past frame using the extracted feature vectors to identify the same object that failed to track. Through the above process, it is possible to re-track the same object that has failed to tracking in the case of re-appearance or occlusion after leaving the video location. As a result, action and facial emotion detection results of the newly recognized object due to the tracking fails can be linked to those of the object that appeared in the past. On the other hand, as a way to improve processing performance, we introduce Bounding Box Queue by Object and Feature Queue method that can reduce RAM memory requirements while maximizing GPU memory throughput. Also we introduce the IoF(Intersection over Face) algorithm that allows facial emotion recognized through AWS Rekognition to be linked with object tracking information. The academic significance of this study is that the two-stage re-identification model can have real-time performance even in a high-cost environment that performs action and facial emotion detection according to processing techniques without reducing the accuracy by using simple metrics to achieve real-time performance. The practical implication of this study is that in various industrial fields that require action and facial emotion detection but have many difficulties due to the fails in object tracking can analyze videos effectively through proposed model. Proposed model which has high accuracy of retrace and processing performance can be used in various fields such as intelligent monitoring, observation services and behavioral or psychological analysis services where the integration of tracking information and extracted metadata creates greate industrial and business value. In the future, in order to measure the object tracking performance more precisely, there is a need to conduct an experiment using the MOT Challenge dataset, which is data used by many international conferences. We will investigate the problem that the IoF algorithm cannot solve to develop an additional complementary algorithm. In addition, we plan to conduct additional research to apply this model to various fields' dataset related to intelligent video analysis.
When a person's face is recognized through a recording device such as a low-pixel surveillance camera, it is difficult to capture the face due to low image quality. In situations where it is difficult to recognize a person's face, problems such as not being able to identify a criminal suspect or a missing person may occur. Existing studies on face recognition used refined datasets, so the performance could not be measured in various environments. Therefore, to solve the problem of poor face recognition performance in low-quality images, this paper proposes a method to generate high-quality images by performing image quality improvement on low-quality facial images considering various environments, and then improve the performance of facial feature point detection. To confirm the practical applicability of the proposed architecture, an experiment was conducted by selecting a data set in which people appear relatively small in the entire image. In addition, by choosing a facial image dataset considering the mask-wearing situation, the possibility of expanding to real problems was explored. As a result of measuring the performance of the feature point detection model by improving the image quality of the face image, it was confirmed that the face detection after improvement was enhanced by an average of 3.47 times in the case of images without a mask and 9.92 times in the case of wearing a mask. It was confirmed that the RMSE for facial feature points decreased by an average of 8.49 times when wearing a mask and by an average of 2.02 times when not wearing a mask. Therefore, it was possible to verify the applicability of the proposed method by increasing the recognition rate for facial images captured in low quality through image quality improvement.
Khori(高麗) refers to the Chaabog(reindeer) that live on lichens(蘚) on Mt. Soyon(鮮) in which pastures are the cold and dry plateau of North Eurasia. Thus, the origin region of the Khori or Koguryo that are the ancestors of the reindeer-herding pastoral nomads(馴鹿 遊牧民) can be said to be the Steppe-Taiga-Tundra pastoral areas of North Eurasia and North America. When the pastoral nomads moved on to the great mountain(大山) zone of the Jangbaek(長白) to the Baekdu(白頭) Mountains, they could have been in contact with pastoral farmers or agricultural farmers living there and they became the farmers remaining on agricultural farms. They were the Koryo people, the ancestors of Korea. Staying in one place, they gradually forgot the origin of their reindeer-herding pastoral nomadic history in the Northwest area of Mt. Soyon, the small mountain(小山) zone of the Steppe-Taiga-Tundra pastoral areas. In other words, they lost their identity as reindeer-herding pastoral nomads when they entered the agricultural area after leaving the pastoral area. However, since their basic genes had already formed when they lived on the cold and dry plateau of North Eurasia, it is possible to study their pastoral nomadic history focusing on 'the minority living in the broad area(廣域少數)', by utilizing highly advanced biotechnological science and focusing on genes and information technology innovation, and removing various past hindrances in research. Therefore, it is not so difficult to restore the reindeerherding pastoral nomadic history of the Koguryo(高句麗) people and secure their pastoral nomadic identity, of which the first steps have already been taken into their historical stages. The Eurasian continent and the Korean peninsula, especially the cold and dry plateau of North Eurasia and the Korean peninsula have been closely related to each other ecologically and historically. They can never be a separate space at all. The Eurasian continent lies horizontally east to west and thus, the continent forms an isothermal zone. Also, since the time of producing their own foods, it was relatively easy for people with their technology to move to other places owing to the pastoral nomadic characteristic of mobility. Unlike the Chungyen(中原) region, western Asia and the regions covering the Siberia-Manchu-Korean peninsula where food production revolution was first made were connected to the Mongolian lichens route(蘚苔之路: Ni, ukinii jam) and steppe roads. Although the ecological conditions of nature have changed a bit throughout a long history, it was natural for the many tribes in North Asia living on the largest Steppe-Taiga-Tundra area in the world to have believed 'the legends related to animals in relation to their founders and ancestors(獸祖傳說)'. Assuming that Siberian tigers and the tigers living on Mt. Baekdu were connected ecologically and genetically because of the ecological characteristics of the animals, and their migration from plateau to plateau, we would suspect that the Chosun(朝鮮) tribe living on Mt. Baekdu were ethnically and culturally more closely connected to the farther removed Ural-Altai tribes that lived on the cold and dry plateau region than to the Han(i14;) tribe who lived in Chungyen(中原) that was close to Mt. Baekdu. More evidence is the structure of the Korean language which has the form of 'Subject + Object + Verb', which is assumed to have originated from the speedy lifestyle of the reindeer-herding pastoral nomads. The structure is quite different from that of the Han(漢) language, which is based on agricultural life. Also, it is natural for reindeer riding reindeerherding pastoral nomads or horse-riding sheep-herding pastoral nomads(騎馬, 羊遊牧民) to have held military and political power over the region and eventually to have established an ancient pastoral nomadic empire in the process of their conquest of agricultural regions. The stages for founding global empires in the history of mankind maybe largely divided into two, in terms of ecological conditions and occupations. They are the steppes and the oceans. Of course, the steppe-based empires were established based on the skills to deal with horses and the ability to shoot arrows while riding horses, along with the use of iron ware in the 8th century BC. The steppe-based empires became the foundation for an oceanic empire, which could have been established by the use of warships and warship guns since the 15th Century. Based on those facts, we know that Chosun, Puyo(夫餘), and Koguryo are the products of a developmental process of pastoral nomadic empires on the steppes. Maybe we can easily find the pastoral nomadic identity of the Koguryo more than we expected when we trace the origins and history of the Korean tribe living in the pastures located in the northwest area of Mt. Jangbaek by focusing on pastoral nomadic mobility and organization just as we have investigated the historic origins of Anglo-Saxons in America by focusing on the times before the 15th Century. In the process, we should keep in mind that English culture originated from the Industrial Revolution and was directly delivered to the American continent, although America was far from England and was not an intermediate point on long sojourns either. Further, American culture came back to England in a more advanced form later. The most important thing currently to be resolved is to cause Koreans to look back on their own history in a freer way of thinking and with diverse, profound, and sharp insight, taking away the old and existing conventional recognition that is entangled with complicated interests with Korean people and other countries. The meanings of Chosun, Khori, and Solongos have been interpreted arbitrarily without any historic evidence by the scholars who followed conventional tradition of fixed-minded aristocrats in an agricultural society. If the Siberian cultural properties of the stone age, the earthenware age, the bronze age, and the iron age are analyzed in such a way, archaeological discovery will never be able to contribute to the restoration of the Koguryo's pastoral nomadic identity. One should transcend the errors that tend to interpret the cultural properties discovered in the pastoral nomadic regions as not being differentiated from those of agricultural regions and just interpret them altogether from the agricultural point of view. A more careful intention is required in the interpretation of cultural properties of ancient Korean empires that seem to have been formed due to mutual interactions of pastoral nomadic and agricultural cultures. Also, it is required that the conventional recognition chain of 'reverse-genes' be severed, which has placed more weight on agricultural properties than pastoral nomadic ones, since their settlement on agricultural farms was made after the establishment of their ancient pastoral nomadic empires. There is no reason at all to place priority on stoneware, earthenware, bronze ware, and iron ware than on wooden ware(木器) and other ware which were made of animal skins(皮器), bones and horns(骨角器), in analyzing the history in the regions of reindeer or sheep pastures. Reading ancient Korean history from the perspective of pastoral nomadic history, one feels strongly the instinctive emotions to return to the natural 'mother place'. The reindeer-herding pastoral nomadic identity of the Koguryo people that has been accumulated in volumes in their genes and hidden deep inside and have interacted organically could be reborn with Burqanism(Burqan refers to 不咸 in Chinese), which was their religion by birth and symbolized as the red willow(紅柳=不咸). The mother place of the Koguryo's people is the endless vast green pastures of North Eurasia and North America, where we anticipated the development of Korean livestock farming following the inherent properties in the genes of the reindeer-herding pastoral nomads with Korean ancestors. We anticipate that the place would be the core resource that could contribute to the development of life of living creatures following the inherent properties of their genes and biotechnological factors. In other words, biotechnology used for a search for clues on the well-being of humans could be the fruit brought by Burqanism of the Koguryo people and the fruit of the globalization of Korean livestock farming. It is the Chosun farmer in China come from the vast nomadic reindeer pastures of North Eurasia that resolved the food problem of a billion Chinese people with lowland paddy rice seeds (水稻) by transforming Heilongjiang Province(黑龍江省) into an oceanic lowland paddy rice field(水田). Even Mao Tse-tung(毛擇東) could not resolve the food problem by his revolution campaigns for tens of years. Today is the very time that requires the development of special livestock farming following the inherent properties of the ancient Korean reindeer-herding pastoral nomads that respected the dignity of life on the cold and dry plateau of North Eurasia and the America continent. I suggest that research should be started from the pastures of the Dariganga Steppe in East Mongolia that was the homeland of Hanwoo(韓牛) and the central horse-herding steppe place(牧馬場) of Chingis Khan's Mongolia. The Dariganga Steppe is awash with an affluent natural environment for pastoral nomadic living however, the quality of life of the pastoral nomads there is still low. I suggest we Koreans, the descendents of the Koguryo, should take our first steps for our livestock farming business project and develop the Northern nomadic pastures, here at the pastures of the Dariganga Steppe, which is the Mongolian core place of state-of-the-art technology for military weapons.