• Title/Summary/Keyword: Multimodal Profile

Search Result 8, Processing Time 0.026 seconds

Multimodal Biometrics Recognition from Facial Video with Missing Modalities Using Deep Learning

  • Maity, Sayan;Abdel-Mottaleb, Mohamed;Asfour, Shihab S.
    • Journal of Information Processing Systems
    • /
    • v.16 no.1
    • /
    • pp.6-29
    • /
    • 2020
  • Biometrics identification using multiple modalities has attracted the attention of many researchers as it produces more robust and trustworthy results than single modality biometrics. In this paper, we present a novel multimodal recognition system that trains a deep learning network to automatically learn features after extracting multiple biometric modalities from a single data source, i.e., facial video clips. Utilizing different modalities, i.e., left ear, left profile face, frontal face, right profile face, and right ear, present in the facial video clips, we train supervised denoising auto-encoders to automatically extract robust and non-redundant features. The automatically learned features are then used to train modality specific sparse classifiers to perform the multimodal recognition. Moreover, the proposed technique has proven robust when some of the above modalities were missing during the testing. The proposed system has three main components that are responsible for detection, which consists of modality specific detectors to automatically detect images of different modalities present in facial video clips; feature selection, which uses supervised denoising sparse auto-encoders network to capture discriminative representations that are robust to the illumination and pose variations; and classification, which consists of a set of modality specific sparse representation classifiers for unimodal recognition, followed by score level fusion of the recognition results of the available modalities. Experiments conducted on the constrained facial video dataset (WVU) and the unconstrained facial video dataset (HONDA/UCSD), resulted in a 99.17% and 97.14% Rank-1 recognition rates, respectively. The multimodal recognition accuracy demonstrates the superiority and robustness of the proposed approach irrespective of the illumination, non-planar movement, and pose variations present in the video clips even in the situation of missing modalities.

A Multimodal Profile Ensemble Approach to Development of Recommender Systems Using Big Data (빅데이터 기반 추천시스템 구현을 위한 다중 프로파일 앙상블 기법)

  • Kim, Minjeong;Cho, Yoonho
    • Journal of Intelligence and Information Systems
    • /
    • v.21 no.4
    • /
    • pp.93-110
    • /
    • 2015
  • The recommender system is a system which recommends products to the customers who are likely to be interested in. Based on automated information filtering technology, various recommender systems have been developed. Collaborative filtering (CF), one of the most successful recommendation algorithms, has been applied in a number of different domains such as recommending Web pages, books, movies, music and products. But, it has been known that CF has a critical shortcoming. CF finds neighbors whose preferences are like those of the target customer and recommends products those customers have most liked. Thus, CF works properly only when there's a sufficient number of ratings on common product from customers. When there's a shortage of customer ratings, CF makes the formation of a neighborhood inaccurate, thereby resulting in poor recommendations. To improve the performance of CF based recommender systems, most of the related studies have been focused on the development of novel algorithms under the assumption of using a single profile, which is created from user's rating information for items, purchase transactions, or Web access logs. With the advent of big data, companies got to collect more data and to use a variety of information with big size. So, many companies recognize it very importantly to utilize big data because it makes companies to improve their competitiveness and to create new value. In particular, on the rise is the issue of utilizing personal big data in the recommender system. It is why personal big data facilitate more accurate identification of the preferences or behaviors of users. The proposed recommendation methodology is as follows: First, multimodal user profiles are created from personal big data in order to grasp the preferences and behavior of users from various viewpoints. We derive five user profiles based on the personal information such as rating, site preference, demographic, Internet usage, and topic in text. Next, the similarity between users is calculated based on the profiles and then neighbors of users are found from the results. One of three ensemble approaches is applied to calculate the similarity. Each ensemble approach uses the similarity of combined profile, the average similarity of each profile, and the weighted average similarity of each profile, respectively. Finally, the products that people among the neighborhood prefer most to are recommended to the target users. For the experiments, we used the demographic data and a very large volume of Web log transaction for 5,000 panel users of a company that is specialized to analyzing ranks of Web sites. R and SAS E-miner was used to implement the proposed recommender system and to conduct the topic analysis using the keyword search, respectively. To evaluate the recommendation performance, we used 60% of data for training and 40% of data for test. The 5-fold cross validation was also conducted to enhance the reliability of our experiments. A widely used combination metric called F1 metric that gives equal weight to both recall and precision was employed for our evaluation. As the results of evaluation, the proposed methodology achieved the significant improvement over the single profile based CF algorithm. In particular, the ensemble approach using weighted average similarity shows the highest performance. That is, the rate of improvement in F1 is 16.9 percent for the ensemble approach using weighted average similarity and 8.1 percent for the ensemble approach using average similarity of each profile. From these results, we conclude that the multimodal profile ensemble approach is a viable solution to the problems encountered when there's a shortage of customer ratings. This study has significance in suggesting what kind of information could we use to create profile in the environment of big data and how could we combine and utilize them effectively. However, our methodology should be further studied to consider for its real-world application. We need to compare the differences in recommendation accuracy by applying the proposed method to different recommendation algorithms and then to identify which combination of them would show the best performance.

Predicting Session Conversion on E-commerce: A Deep Learning-based Multimodal Fusion Approach

  • Minsu Kim;Woosik Shin;SeongBeom Kim;Hee-Woong Kim
    • Asia pacific journal of information systems
    • /
    • v.33 no.3
    • /
    • pp.737-767
    • /
    • 2023
  • With the availability of big customer data and advances in machine learning techniques, the prediction of customer behavior at the session-level has attracted considerable attention from marketing practitioners and scholars. This study aims to predict customer purchase conversion at the session-level by employing customer profile, transaction, and clickstream data. For this purpose, we develop a multimodal deep learning fusion model with dynamic and static features (i.e., DS-fusion). Specifically, we base page views within focal visist and recency, frequency, monetary value, and clumpiness (RFMC) for dynamic and static features, respectively, to comprehensively capture customer characteristics for buying behaviors. Our model with deep learning architectures combines these features for conversion prediction. We validate the proposed model using real-world e-commerce data. The experimental results reveal that our model outperforms unimodal classifiers with each feature and the classical machine learning models with dynamic and static features, including random forest and logistic regression. In this regard, this study sheds light on the promise of the machine learning approach with the complementary method for different modalities in predicting customer behaviors.

Estimation of Doppler Spectrum Modes in a Weather Radar for Detection of Hazardous Weather Conditions

  • Lee, Jong-Gil
    • The Journal of Korean Institute of Communications and Information Sciences
    • /
    • v.27 no.3A
    • /
    • pp.205-210
    • /
    • 2002
  • In a Doppler weather radar, high resolution windspeed profile measurements are needed to provide the reliable detection of hazardous weather conditions. For this purpose, the pulse pair method is generally considered to be the most efficient estimator, However, this estimator has some bias errors due to asymmetric spectra and may yield meaningless results in the case of a multimodal return spectrum. Although the poly-pulse pair method can reduce the bias errors of skewed weather spectra, the modes of spectrum may provide more reliable information than the statistical mean for the case of a multimodal or seriously skewed spectrum. Therefore, the idea of relatively simple mode estimator for a weather radar is developed in this paper, Performance simulations show promising results in the detection of hazardous weather conditions.

Buffeting response of a free-standing bridge pylon in a trumpet-shaped mountain pass

  • Li, Jiawu;Shen, Zhengfeng;Xing, Song;Gao, Guangzhong
    • Wind and Structures
    • /
    • v.30 no.1
    • /
    • pp.85-97
    • /
    • 2020
  • The accurate estimation of the buffeting response of a bridge pylon is related to the quality of the bridge construction. To evaluate the influence of wind field characteristics on the buffeting response of a pylon in a trumpet-shaped mountain pass, this paper deduced a multimodal coupled buffeting frequency domain calculation method for a variable-section bridge tower under the twisted wind profile condition based on quasi-steady theory. Through the long-term measurement of the wind field of the trumpet-shaped mountain pass, the wind characteristics were studied systematically. The effects of the wind characteristics, wind yaw angles, mean wind speeds, and wind profiles on the buffeting response were discussed. The results show that the mean wind characteristics are affected by the terrain and that the wind profile is severely twisted. The optimal fit distribution of the monthly and annual maximum wind speeds is the log-logistic distribution, and the generalized extreme value I distribution may underestimate the return wind speed. The design wind characteristics will overestimate the buffeting response of the pylon. The buffeting response of the pylon is obviously affected by the wind yaw angle and mean wind speed. To accurately estimate the buffeting response of the pylon in an actual construction, it is necessary to consider the twisted effect of the wind profile.

A Study on the Detection of Hazardous Weather Conditions by a Doppler Weather Radar (도플러 레이다를 이용한 기상위험 탐지에 관한 연구)

  • 이종길
    • The Journal of Korean Institute of Communications and Information Sciences
    • /
    • v.19 no.3
    • /
    • pp.533-542
    • /
    • 1994
  • In a Doppler weather radar, high resolution windspeed profile measurements are needed to provide reliable detection of a hazardous weather condition. For this purpose, the pulse-pair method is generally considered to be the most efficient estimator. However, this estimator has some bias errors due to asymmetric spectra and may yield meaningless results in the case of a multimodal return spectrum in this paper, bias errors were analyzed and an improved method was suggested to reduece these errors. For the case of a multimodal or seriously skewed spectrum, the modes of spectrum may provide more reliable information than the statistical mean. Therefore, the idea of a relatively simple mode estimator is also developed.

  • PDF

Oxaliplatin and Leucovorin Plus Fluorouracil Combination Chemotherapy as a First-line versus Salvage Treatment in HER2-negative Advanced Gastric Cancer Patients

  • Hee Seok Moon;Jae Ho Park;Ju Seok Kim;Sun Hyung Kang;Jae Kyu Seong;Hyun Yong Jeong
    • Journal of Digestive Cancer Research
    • /
    • v.6 no.1
    • /
    • pp.25-31
    • /
    • 2018
  • Background: In Korea, stomach cancer is the second most common malignancy and the third leading cause of cancer-related deaths. the time of diagnosis is very important for treatment so early detection and surgery are currently considered the mainstay of treatment, when diagnosed advanced with tumor extension through the gastric wall and direct extension into other organs, with metastatic involvement. Recently, new drugs, drug combinations, and multimodal approaches have been used to treat this disease and In cancers over expressing or amplifying HER2, the combination of cisplatin-fluoropyrimidine-trastuzumab is considered to be the treatment of reference. but At present, the choice of treatment schedule for HER2-negative tumors is based on the medical institution's preferences and adverse effects profile. The aim of this study was to evaluate the effectiveness and safety of using FOLFOX regimen as a first-line therapy or a salvage therapy in the patients with HER2-negative advanced or metastatic gastric cancer. Methods: We retrospective reviewed the patient medical record from March 2012 to July 2017. This study evaluated 113 patients. Sixty-eight patients were treated with the FOLFOX regimen for the first time (first-line group) and 45 patients were treated with the FOLFOX regimen as a second (35 patients) or third (10 patients) chemotherapy (salvage group). Results: In the first-line group, the response rate was 54.9%. In the salvage therapy group, the response rate was 24.4% and The difference was statistically significant (p=0.205). The median TTP of the first-line group was 10.7 months (95% confidence interval [95% CI], 7.8-13.7 months) and that of salvage line group was 6.1 months (95% CI, 3.8-8.4 months). The median OS of the first-line group was 15.8 months (95% CI, 12.7-18.9 months) and that of the salvage therapy group was 10.2 months (95% CI, 8.2-11.9 months). drug toxicity was similar andtolerable between two groups. Conclusion: In patients with unresctable metastatic gastric cancer, after failing to respond to first-line therapy, most patients have no alternative other than second-line therapy because the disease is highly progressive. if the performance status of the patient is good enough to be eligible to treatments beyond best supportive care. FOLFOX regimen can be a considerable therapeutic option for salvage treatment.

  • PDF

Tidal-Flat Sedimentation in a Semienclosed Bay with Erosional Shorelines: Hampyong Bay, West Coast of Korea (해안침식이 우세한 반폐쇄적 조간대의 퇴적작용: 한국 서해안의 함평만)

  • Chang, Jin-Ho;Kim, Yeo-Sang;Cho, Yeong-Gil
    • The Sea:JOURNAL OF THE KOREAN SOCIETY OF OCEANOGRAPHY
    • /
    • v.4 no.2
    • /
    • pp.117-126
    • /
    • 1999
  • Hampyong Bay is a semienclosed and macrotidal bay which opens to the eastern Yellow Sea through a narrow inlet in the southwestern coast of Korea. In order to understand the tidal-flat sedimentation in the semienclosed setting, morphology, sediments, accumulation rate and sea cliff erosion were investigated in the tidal flat of Hampyong Bay. The tidal flat of Hampyong Bay lacks intertidal drainage systems, and generally shows the concave-upward profile whose relief is designated by marked morphological features such as high-tide beaches, intertidal sand shoals and tidal creeks. Surfacial sediments of the tidal flat mainly consist of mud, sandy mud, gravelly mud, gravelly sand and muddy gravel, thus showing the textural characteristics of multimodal grain-size distribution, poorly sorting and positive skewness. The sediments generally coarsen landward due to the increase in coarse fraction content. Sedimentary structures are deeply bioturbated, but parallel lamination and lenticular bedding are locally found in the mudflat near mean low water line. Annual accumulation rates across the tidal flat (along Line SM) average -5.2 cm/yr with a range of -45.8~+4.2 cm/yr, indicating that the tidal flat is erosional. In general, erosion rates of upper and lower tidal flat are higher than those of middle tidal flat. Seasonally, the erosion rates are much higher during spring and winter when dominant wind direction corresponds to the long axis of Hampyong Bay. Sea cliffs are eroded at a rate of 1.4 m/yr. The biggest sea cliff erosion generally occurs 1~2 months later after tidal flats were extensively eroded. Such erosions of tidal Oats and sea cliffs in the semienclosed bay setting are interpreted to be due to wind waves coupled with local sea-level rise.

  • PDF