• Title/Summary/Keyword: Multimodal studies

Search Result 83, Processing Time 0.026 seconds

Emotion Recognition Implementation with Multimodalities of Face, Voice and EEG

  • Udurume, Miracle;Caliwag, Angela;Lim, Wansu;Kim, Gwigon
    • Journal of information and communication convergence engineering
    • /
    • v.20 no.3
    • /
    • pp.174-180
    • /
    • 2022
  • Emotion recognition is an essential component of complete interaction between human and machine. The issues related to emotion recognition are a result of the different types of emotions expressed in several forms such as visual, sound, and physiological signal. Recent advancements in the field show that combined modalities, such as visual, voice and electroencephalography signals, lead to better result compared to the use of single modalities separately. Previous studies have explored the use of multiple modalities for accurate predictions of emotion; however the number of studies regarding real-time implementation is limited because of the difficulty in simultaneously implementing multiple modalities of emotion recognition. In this study, we proposed an emotion recognition system for real-time emotion recognition implementation. Our model was built with a multithreading block that enables the implementation of each modality using separate threads for continuous synchronization. First, we separately achieved emotion recognition for each modality before enabling the use of the multithreaded system. To verify the correctness of the results, we compared the performance accuracy of unimodal and multimodal emotion recognitions in real-time. The experimental results showed real-time user emotion recognition of the proposed model. In addition, the effectiveness of the multimodalities for emotion recognition was observed. Our multimodal model was able to obtain an accuracy of 80.1% as compared to the unimodality, which obtained accuracies of 70.9, 54.3, and 63.1%.

Relationship between Postural Balance Training and Fall Risks for Elderly: a Systematic Review of Randomized Controlled Trials

  • Kim, Heesuk;Hwang, Sujin
    • Physical Therapy Rehabilitation Science
    • /
    • v.10 no.2
    • /
    • pp.185-196
    • /
    • 2021
  • Objective: Falling is one of main accident to facilitate the physical injuries in order adults. The purpose of the systematic review was to determine the effects of postural balance training whether the recovery of falls in elderly with normal physical function or not throughout summing the selected studies quantitatively. Design: A systematic review Methods: MEDLINE and other four databases were searched up to April 20, 2021 and randomized controlled trials (RCTs) evaluating postural balance approaches on fall risks in elderly. The researched studies excluded the double studies, titles and abstract, and finally full-reported study. The selected RCTs studies were extracted characteristics of the studies and summary of results based on PICOS-SD (population, intervention, comparison, outcomes, and setting- study design) model to synthesize the papers qualitatively. Results: The review involved 22 RCT reports with 4,847 community older adults aged 65 years or over. Nineteen of the selected RCT studies reported dual or multimodal exercises show the beneficial effect for older adults compared to one-type treatment or no intervention. All of selected showed low risk in the selection, attrition, and reporting bias. However, detection bias showed low risk at 75% records of the involved RCTs and performance bias was low risk at only three records. Conclusions: The results of the systematic review propose that a standardized therapeutic approach and the intensity are needed for improving risk of falls in older adults.

An Intelligent Emotion Recognition Model Using Facial and Bodily Expressions

  • Jae Kyeong Kim;Won Kuk Park;Il Young Choi
    • Asia pacific journal of information systems
    • /
    • v.27 no.1
    • /
    • pp.38-53
    • /
    • 2017
  • As sensor technologies and image processing technologies make collecting information on users' behavior easy, many researchers have examined automatic emotion recognition based on facial expressions, body expressions, and tone of voice, among others. Specifically, many studies have used normal cameras in the multimodal case using facial and body expressions. Thus, previous studies used a limited number of information because normal cameras generally produce only two-dimensional images. In the present research, we propose an artificial neural network-based model using a high-definition webcam and Kinect to recognize users' emotions from facial and bodily expressions when watching a movie trailer. We validate the proposed model in a naturally occurring field environment rather than in an artificially controlled laboratory environment. The result of this research will be helpful in the wide use of emotion recognition models in advertisements, exhibitions, and interactive shows.

A Review on Brain Imaging Studies of Suicide in Youth (청소년기 자살에 대한 뇌영상 연구)

  • Lee, Suji;Kim, Shinhye;Yoon, Sujung
    • Korean Journal of Biological Psychiatry
    • /
    • v.28 no.2
    • /
    • pp.36-49
    • /
    • 2021
  • Suicide is a leading cause of death worldwide, especially among adolescents and young adults. Considering this fact, it is imperative that we understand the neural mechanisms underlying suicidal thoughts and behaviors in youth from a neurodevelopmental perspective. In this review, we focused on the magnetic resonance imaging studies that examined the neural correlates of suicidal ideations (SI) or attempts (SA) in youth. We reviewed twenty-three cross-sectional studies reporting the structural and functional alterations in association with SI or SA among adolescents and young adults with various mental disorders. The previous literature suggests that the dorsolateral prefrontal cortex, anterior cingulate cortex, and ventral frontolimbic circuit, may play an important role in the pathophysiology of suicidal behavior in youth through altered top-down control over emotion and impulsivity. Future studies with a longitudinal design and using multimodal imaging techniques may be of help to identify novel therapeutic targets specific for youth with suicidal thoughts and behaviors.

Development of a multi-modal imaging system for single-gamma and fluorescence fusion images

  • Young Been Han;Seong Jong Hong;Ho-Young Lee;Seong Hyun Song
    • Nuclear Engineering and Technology
    • /
    • v.55 no.10
    • /
    • pp.3844-3853
    • /
    • 2023
  • Although radiation and chemotherapy methods for cancer therapy have advanced significantly, surgical resection is still recommended for most cancers. Therefore, intraoperative imaging studies have emerged as a surgical tool for identifying tumor margins. Intraoperative imaging has been examined using conventional imaging devices, such as optical near-infrared probes, gamma probes, and ultrasound devices. However, each modality has its limitations, such as depth penetration and spatial resolution. To overcome these limitations, hybrid imaging modalities and tracer studies are being developed. In a previous study, a multi-modal laparoscope with silicon photo-multiplier (SiPM)-based gamma detection acquired a 1 s interval gamma image. However, improvements in the near-infrared fluorophore (NIRF) signal intensity and gamma image central defects are needed to further evaluate the usefulness of multi-modal systems. In this study, an attempt was made to change the NIRF image acquisition method and the SiPM-based gamma detector to improve the source detection ability and reduce the image acquisition time. The performance of the multi-modal system using a complementary metal oxide semiconductor and modified SiPM gamma detector was evaluated in a phantom test. In future studies, a multi-modal system will be further optimized for pilot preclinical studies.

A Literature Review of Aromatherapy Used in Stress Relief (스트레스 완화 목적의 아로마 요법에 관한 문헌고찰)

  • Kim, Hyeon-Jin;Jeong, Soo-Hyun;Jeong, Hye-In;Kim, Kyeong Han
    • Journal of Society of Preventive Korean Medicine
    • /
    • v.25 no.2
    • /
    • pp.45-60
    • /
    • 2021
  • Objective : This study was aimed to review randomized controlled trials (RCTs) about whether aromatherapy relieves stress. Method : We searched document about criteria to use words like 'Aroma', 'Oil' and 'Stress'. The study included 24 RCTs which were selected by total 167 studies searched in Korean Journal by searching OASIS, ScienceON, KISS, RISS. Cases that cannot be performed alone are excluded. Results : We got 24 domestic standard documents. Of the 24 studies, 14 were for students, and 6 were for patients receiving hospital treatment. Among the 7 treatments, dry-inhalation was used 13 times, and necklace-inhalation was used 9 times. Of the 24 Studies, lavender oil was used 19 times and sweet orange was used 4 times. Among the 28 types of measuring instruments used, 10 related to the autonomic nervous system and 8 STAIs and VASs were used respectively. Conclusion : It was possible to conclude that aromatherapy was effective in relieving stress. Through further research, it is necessary to study effective oil mixing methods, methods for measuring subjective stress, multimodal intervention, and effective intervention periods.

A Multimodal Profile Ensemble Approach to Development of Recommender Systems Using Big Data (빅데이터 기반 추천시스템 구현을 위한 다중 프로파일 앙상블 기법)

  • Kim, Minjeong;Cho, Yoonho
    • Journal of Intelligence and Information Systems
    • /
    • v.21 no.4
    • /
    • pp.93-110
    • /
    • 2015
  • The recommender system is a system which recommends products to the customers who are likely to be interested in. Based on automated information filtering technology, various recommender systems have been developed. Collaborative filtering (CF), one of the most successful recommendation algorithms, has been applied in a number of different domains such as recommending Web pages, books, movies, music and products. But, it has been known that CF has a critical shortcoming. CF finds neighbors whose preferences are like those of the target customer and recommends products those customers have most liked. Thus, CF works properly only when there's a sufficient number of ratings on common product from customers. When there's a shortage of customer ratings, CF makes the formation of a neighborhood inaccurate, thereby resulting in poor recommendations. To improve the performance of CF based recommender systems, most of the related studies have been focused on the development of novel algorithms under the assumption of using a single profile, which is created from user's rating information for items, purchase transactions, or Web access logs. With the advent of big data, companies got to collect more data and to use a variety of information with big size. So, many companies recognize it very importantly to utilize big data because it makes companies to improve their competitiveness and to create new value. In particular, on the rise is the issue of utilizing personal big data in the recommender system. It is why personal big data facilitate more accurate identification of the preferences or behaviors of users. The proposed recommendation methodology is as follows: First, multimodal user profiles are created from personal big data in order to grasp the preferences and behavior of users from various viewpoints. We derive five user profiles based on the personal information such as rating, site preference, demographic, Internet usage, and topic in text. Next, the similarity between users is calculated based on the profiles and then neighbors of users are found from the results. One of three ensemble approaches is applied to calculate the similarity. Each ensemble approach uses the similarity of combined profile, the average similarity of each profile, and the weighted average similarity of each profile, respectively. Finally, the products that people among the neighborhood prefer most to are recommended to the target users. For the experiments, we used the demographic data and a very large volume of Web log transaction for 5,000 panel users of a company that is specialized to analyzing ranks of Web sites. R and SAS E-miner was used to implement the proposed recommender system and to conduct the topic analysis using the keyword search, respectively. To evaluate the recommendation performance, we used 60% of data for training and 40% of data for test. The 5-fold cross validation was also conducted to enhance the reliability of our experiments. A widely used combination metric called F1 metric that gives equal weight to both recall and precision was employed for our evaluation. As the results of evaluation, the proposed methodology achieved the significant improvement over the single profile based CF algorithm. In particular, the ensemble approach using weighted average similarity shows the highest performance. That is, the rate of improvement in F1 is 16.9 percent for the ensemble approach using weighted average similarity and 8.1 percent for the ensemble approach using average similarity of each profile. From these results, we conclude that the multimodal profile ensemble approach is a viable solution to the problems encountered when there's a shortage of customer ratings. This study has significance in suggesting what kind of information could we use to create profile in the environment of big data and how could we combine and utilize them effectively. However, our methodology should be further studied to consider for its real-world application. We need to compare the differences in recommendation accuracy by applying the proposed method to different recommendation algorithms and then to identify which combination of them would show the best performance.

Effects of a Short-term Multimodal Group Intervention Program on Cognitive Function and Depression of the Elderly (단기 집단 복합중재가 정상 노인의 인지기능 및 우울에 미치는 영향)

  • Jung, Beom-Jin;Choi, Yu-Jin
    • Therapeutic Science for Rehabilitation
    • /
    • v.8 no.3
    • /
    • pp.57-68
    • /
    • 2019
  • Purpose: This study aimed to investigate the effects of a short-term group multimodal intervention program that mixes physical activity, cognitive motion, and social interaction, on the cognitive function and depression level of healthy over 75-year-old individuals. Method: This study used a one group pre-test-post-test design, and intervention was made for 70 minutes per session, once a week, for four sessions in total. To compare changes in cognitive function, depression level and physical function before and after intervention, this study used the Mini-Mental State Examination-Dementia Screening (MMSE-DS), Geriatric Depression Scale-Short Form (GDS-SF), and Berg Balance Scale (BBS). Result: After applying group multimodal interventions to healthy over 75-year-old individuals, there was a statistically significant improvement in their cognitive function (p < 0.01), and there was a statistically significant decrease in their depression level (p < 0.05). Also, there was an increase in the rating score of the degree of balance from $46.83{\pm}9.11$ points before the intervention, to $48.08{\pm}7.00$ points after the intervention; however, it was not statistically significant (p > 0.05). Conclusion: Short-term group multimodal intervention that mixes physical activity, cognitive motion, and social interaction had a significant effect on slowing down the deterioration of cognitive function in healthy over 75 year-old individuals, and decreased their depression level. This study is significant in that it presents a foundation for providing more systematic intervention for the prevention of dementia and depression in the healthy older individuals. Follow-up studies should verify the result through research on the effects of an occupational therapist's professional treatment, and experimental group-control research.

Accuracy Evaluation of Three-Dimensional Multimodal Image Registration Using a Brain Phantom (뇌팬톰을 이용한 삼차원 다중영상정합의 정확성 평가)

  • 진호상;송주영;주라형;정수교;최보영;이형구;서태석
    • Journal of Biomedical Engineering Research
    • /
    • v.25 no.1
    • /
    • pp.33-41
    • /
    • 2004
  • Accuracy of registration between images acquired from various medical image modalities is one of the critical issues in radiation treatment planing. In this study, a method of accuracy evaluation of image registration using a homemade brain phantom was investigated. Chamfer matching of CT-MR and CT-SPECT imaging was applied for the multimodal image registration. The accuracy of image correlation was evaluated by comparing the center points of the inserted targets of the phantom. The three dimensional root-mean-square translation deviations of the CT-MR and CT-SPECT registration were 2.1${\pm}$0.8 mm and 2.8${\pm}$1.4 mm, respectively. The rotational errors were < 2$^{\circ}$ for the three orthogonal axes. These errors were within a reasonable margin compared with the previous phantom studies. A visual inspection of the superimposed CT-MR and CT- SPECT images also showed good matching results.

Deep Multimodal MRI Fusion Model for Brain Tumor Grading (뇌 종양 등급 분류를 위한 심층 멀티모달 MRI 통합 모델)

  • Na, In-ye;Park, Hyunjin
    • Proceedings of the Korean Institute of Information and Commucation Sciences Conference
    • /
    • 2022.05a
    • /
    • pp.416-418
    • /
    • 2022
  • Glioma is a type of brain tumor that occurs in glial cells and is classified into two types: high hrade hlioma with a poor prognosis and low grade glioma. Magnetic resonance imaging (MRI) as a non-invasive method is widely used in glioma diagnosis research. Studies to obtain complementary information by combining multiple modalities to overcome the incomplete information limitation of single modality are being conducted. In this study, we developed a 3D CNN-based model that applied input-level fusion to MRI of four modalities (T1, T1Gd, T2, T2-FLAIR). The trained model showed classification performance of 0.8926 accuracy, 0.9688 sensitivity, 0.6400 specificity, and 0.9467 AUC on the validation data. Through this, it was confirmed that the grade of glioma was effectively classified by learning the internal relationship between various modalities.

  • PDF