• Title/Summary/Keyword: Quality evaluation metrics

Search Result 200, Processing Time 0.029 seconds

KAB: Knowledge Augmented BERT2BERT Automated Questions-Answering system for Jurisprudential Legal Opinions

  • Alotaibi, Saud S.;Munshi, Amr A.;Farag, Abdullah Tarek;Rakha, Omar Essam;Al Sallab, Ahmad A.;Alotaibi, Majid
    • International Journal of Computer Science & Network Security
    • /
    • v.22 no.6
    • /
    • pp.346-356
    • /
    • 2022
  • The jurisprudential legal rules govern the way Muslims react and interact to daily life. This creates a huge stream of questions, that require highly qualified and well-educated individuals, called Muftis. With Muslims representing almost 25% of the planet population, and the scarcity of qualified Muftis, this creates a demand supply problem calling for Automation solutions. This motivates the application of Artificial Intelligence (AI) to solve this problem, which requires a well-designed Question-Answering (QA) system to solve it. In this work, we propose a QA system, based on retrieval augmented generative transformer model for jurisprudential legal question. The main idea in the proposed architecture is the leverage of both state-of-the art transformer models, and the existing knowledge base of legal sources and question-answers. With the sensitivity of the domain in mind, due to its importance in Muslims daily lives, our design balances between exploitation of knowledge bases, and exploration provided by the generative transformer models. We collect a custom data set of 850,000 entries, that includes the question, answer, and category of the question. Our evaluation methodology is based on both quantitative and qualitative methods. We use metrics like BERTScore and METEOR to evaluate the precision and recall of the system. We also provide many qualitative results that show the quality of the generated answers, and how relevant they are to the asked questions.

An AutoML-driven Antenna Performance Prediction Model in the Autonomous Driving Radar Manufacturing Process

  • So-Hyang Bak;Kwanghoon Pio Kim
    • KSII Transactions on Internet and Information Systems (TIIS)
    • /
    • v.17 no.12
    • /
    • pp.3330-3344
    • /
    • 2023
  • This paper proposes an antenna performance prediction model in the autonomous driving radar manufacturing process. Our research work is based upon a challenge dataset, Driving Radar Manufacturing Process Dataset, and a typical AutoML machine learning workflow engine, Pycaret open-source Python library. Note that the dataset contains the total 70 data-items, out of which 54 used as input features and 16 used as output features, and the dataset is properly built into resolving the multi-output regression problem. During the data regression analysis and preprocessing phase, we identified several input features having similar correlations and so detached some of those input features, which may become a serious cause of the multicollinearity problem that affect the overall model performance. In the training phase, we train each of output-feature regression models by using the AutoML approach. Next, we selected the top 5 models showing the higher performances in the AutoML result reports and applied the ensemble method so as for the selected models' performances to be improved. In performing the experimental performance evaluation of the regression prediction model, we particularly used two metrics, MAE and RMSE, and the results of which were 0.6928 and 1.2065, respectively. Additionally, we carried out a series of experiments to verify the proposed model's performance by comparing with other existing models' performances. In conclusion, we enhance accuracy for safer autonomous vehicles, reduces manufacturing costs through AutoML-Pycaret and machine learning ensembled model, and prevents the production of faulty radar systems, conserving resources. Ultimately, the proposed model holds significant promise not only for antenna performance but also for improving manufacturing quality and advancing radar systems in autonomous vehicles.

Development of Evaluation Indicators and Usability Evaluation of Kiosk for the Elderly - the Case of KORAIL's Kiosk for Ticketing (장·노년층 대상 키오스크 사용성 측정 지표 개발 및 사용성 평가 - 코레일 열차 발권 키오스크 개발 사례)

  • Sin, Eun-joo;Lim, Soon-Bum
    • The Journal of the Korea Contents Association
    • /
    • v.22 no.1
    • /
    • pp.188-196
    • /
    • 2022
  • Although the use of kiosks is increasing recently, the digital divide for the elderly who uses them is not decreasing. We live in an environment where the use of kiosks is becoming a necessity rather than an option. In such an environment, the digital alienation of the elderly is becoming a problem directly related to the quality of life. Even if a kiosk is developed considering the elderly, the verification of its effectiveness is ambiguous, or in most cases, it depends on the designer's experiential ability rather than the consideration of usability. In this study, the usability of the kiosk was analyzed for the development of kiosk contents for the elderly. The metrics were defined as availability, usefulness, efficiency, attractiveness, and visibility. And the measurement method of the measurement index was developed, and the usability of the kiosk for the elderly was confirmed by performing usability evaluation. This is a method to verify whether the kiosk in the development process can support the elderly or whether the improved kiosk actually increases the usability of the elderly. As a result, it is expected to contribute to improving the accessibility of the kiosk.

One-shot multi-speaker text-to-speech using RawNet3 speaker representation (RawNet3를 통해 추출한 화자 특성 기반 원샷 다화자 음성합성 시스템)

  • Sohee Han;Jisub Um;Hoirin Kim
    • Phonetics and Speech Sciences
    • /
    • v.16 no.1
    • /
    • pp.67-76
    • /
    • 2024
  • Recent advances in text-to-speech (TTS) technology have significantly improved the quality of synthesized speech, reaching a level where it can closely imitate natural human speech. Especially, TTS models offering various voice characteristics and personalized speech, are widely utilized in fields such as artificial intelligence (AI) tutors, advertising, and video dubbing. Accordingly, in this paper, we propose a one-shot multi-speaker TTS system that can ensure acoustic diversity and synthesize personalized voice by generating speech using unseen target speakers' utterances. The proposed model integrates a speaker encoder into a TTS model consisting of the FastSpeech2 acoustic model and the HiFi-GAN vocoder. The speaker encoder, based on the pre-trained RawNet3, extracts speaker-specific voice features. Furthermore, the proposed approach not only includes an English one-shot multi-speaker TTS but also introduces a Korean one-shot multi-speaker TTS. We evaluate naturalness and speaker similarity of the generated speech using objective and subjective metrics. In the subjective evaluation, the proposed Korean one-shot multi-speaker TTS obtained naturalness mean opinion score (NMOS) of 3.36 and similarity MOS (SMOS) of 3.16. The objective evaluation of the proposed English and Korean one-shot multi-speaker TTS showed a prediction MOS (P-MOS) of 2.54 and 3.74, respectively. These results indicate that the performance of our proposed model is improved over the baseline models in terms of both naturalness and speaker similarity.

A Wireless Traffic Load-Balancing Algorithm based on Adaptive Bandwidth Reservation Scheme in Mobile Cellular Networks (셀룰러 망에서 적응적 대역폭 예약 기법을 이용한 무선 트래픽 부하 균형 알고리즘)

  • 정영석;우매리;김종근
    • Proceedings of the Korea Institute of Convergence Signal Processing
    • /
    • 2001.06a
    • /
    • pp.21-24
    • /
    • 2001
  • For very large multimedia traffic to be supported successfully in wireless network environment, it is necessary to provide Quality-of-Service(QoS) guarantees between mobile hosts(clients). In order to guarantee the Qos, we have to keep the call blocking probability below target value during handoff session. However, the QoS negotiated between the client and the network may not be guaranteed due to lack of available channels for traffic in the new cell, since mobile clients should be able to continue their on-going sessions. In this paper we propose a efficient load-balancing algorithm based on the adaptive bandwidth reservation scheme for enlarging available channels in a cell. We design a new method to predict the mobility of clients using MPT(mobility profile table). This method is then used to reserve a part of bandwidths for handoff calls to its adjacent cells and this reserved bandwidth can be used for handoff call prior to new connection requests. If the number of free channels is also under a low threshold value, our scheme use a load-balancing algorithm with a adaptive bandwidth reservation. In order to evaluate the performance of our algorithm, we measure the metrics such as the blocking probability of new calls and dropping probability of handoff calls, and compare with other existing schemes.

  • PDF

Incentive Design Considerations for Free-riding Prevention in Cooperative Distributed Systems (협조적 분산시스템 환경에서 무임승차 방지를 위한 인센티브 디자인 고려사항 도출에 관한 연구)

  • Shin, Kyu-Yong;Yoo, Jin-Cheol;Lee, Jong-Deog;Park, Byoung-Chul
    • Journal of the Korea Society of Computer and Information
    • /
    • v.16 no.7
    • /
    • pp.137-148
    • /
    • 2011
  • Different from the traditional client-server model, it is possible for participants in a cooperative distributed system to get quality services regardless of the number of participants in the system since they voluntarily pool or share their resources in order to achieve their common goal. However, some selfish participants try to avoid providing their resources while still enjoying the benefits offered by the system, which is termed free-riding. The results of free-riding in cooperative distributed systems lead to system collapse because the system capacity (per participant) decreases as the number of free-riders increases, widely known as the tragedy of commons. As a consequence, designing an efficient incentive mechanism to prevent free-riding is mandatory for a successful cooperative distributed system. Because of the importance of incentive mechanisms in cooperative distributed system, a myriad of incentives mechanisms have been proposed without a standard for performance evaluation. This paper draws general incentive design considerations which can be used as performance metrics through an extensive survey on this literature, providing future researchers with guidelines for the effective incentive design in cooperative distributed systems.

A Study on the Measurement Method of Cold Chain Service Quality Using Smart Contract of Blockchain (블록체인의 스마트계약을 이용한 콜드체인 서비스 품질 측정 방안에 대한 연구)

  • Kim, ChangHyun;Shin, KwangSup
    • The Journal of Society for e-Business Studies
    • /
    • v.24 no.3
    • /
    • pp.1-18
    • /
    • 2019
  • Due to the great advances in e-Marketplace and changes in type of items purchased from the online market, it has been dramatically increased the demand of the storage and transportation under the special conditions such as restricted temperature. Especially, the cold chain needs the way to transparently measure and monitor the entire network in realtime because it has a very complicated structure and requires totally different criteria at the every different steps and items. In this research, it has been presented the performance evaluation metrics to make contract using service level agreement (SLA), the way to apply the smart contract based on blockchain, the structure of blocks, service platform and application in order to build cold chain which can prevent the risk factors by measuring and sharing information in realtime using block chain technology. In addition, we have proposed the way to store the measured performance and reputation of each player in the block using smart contract based on SLA. With the presented framework, all players including service providers as well as users can secure the information for making the rational decisions. When the service platform is actually built and operated, it seems possible to secure the information in transparently and realtime. Also, it is possible to prevent the risk factors or prepare the preemptive plans to react on them.

Group-based Adaptive Rendering for 6DoF Immersive Video Streaming (6DoF 몰입형 비디오 스트리밍을 위한 그룹 분할 기반 적응적 렌더링 기법)

  • Lee, Soonbin;Jeong, Jong-Beom;Ryu, Eun-Seok
    • Journal of Broadcast Engineering
    • /
    • v.27 no.2
    • /
    • pp.216-227
    • /
    • 2022
  • The MPEG-I (Immersive) group is working on a standardization project for immersive video that provides 6 degrees of freedom (6DoF). The MPEG Immersion Video (MIV) standard technology is intended to provide limited 6DoF based on depth map-based image rendering (DIBR) technique. Many efficient coding methods have been suggested for MIV, but efficient transmission strategies have received little attention in MPEG-I. This paper proposes group-based adaptive rendering method for immersive video streaming. Each group can be transmitted independently using group-based encoding, enabling adaptive transmission depending on the user's viewport. In the rendering process, the proposed method derives weights of group for view synthesis and allocate high quality bitstream according to a given viewport. The proposed method is implemented through the Test Model for Immersive Video (TMIV) test model. The proposed method demonstrates 17.0% Bjontegaard-delta rate (BD-rate) savings on the peak signalto-noise ratio (PSNR) and 14.6% on the Immersive Video PSNR(IV-PSNR) in terms of various end-to-end evaluation metrics in the experiment.

Intergrated Ecological Health Assessments in Cho River (초강의 통합적 생태건강성 평가)

  • Choi, Ji-Woong;An, Kwang-Guk
    • Korean Journal of Ecology and Environment
    • /
    • v.39 no.3 s.117
    • /
    • pp.320-330
    • /
    • 2006
  • An integrated health of a lotic ecosystem, Cho River, was evaluated by various approaches such as conventional water quality analysis, physical assessments of Qualitative Habitat Evaluation Index (QHEI), and the bioassay of Index of Biological Integrity (IBI) durin August${\sim}$September 2005. The IBI model used in the study was based on original multivariate metric model and then modified the metric attributes of the model for the regional application. Physical habitat health, based on the QHEI, was estimated using eleven metrics. During the study, values of IBI model averaged 36, which was judged as 'fair' to 'good' conditions. Spatial variations in the model values were evident: the headwater site (S1) was estimated as 48, indicating an 'excellent' condition, and the other sites were estimated 32${\sim}$38, 'good' condition. Values of the QHEI in the all sites averaged 148, which is judged as a good condition. The QHEI values varied from 120 (fair condition) to 199 (excellent condition) depending on the location of the stream. Site 5 (S5) was estimated as 'fair${\sim}$good' condition, while Site 7 (S7) was estimated as 'excellent' condition. The biological health, based on the IBI, reflected the habitat health. However, chemical conditions in terms of pH, turbidity, electric conductivity, dissolved oxygen (DO) did not make a difference in the biological health because of minor chemical differences among the locations.

Target candidate fish species selection method based on ecological survey for hazardous chemical substance analysis (유해화학물질 분석을 위한 생태조사 기반의 타깃 후보어종 선정법)

  • Ji Yoon Kim;Sang-Hyeon Jin;Min Jae Cho;Hyeji Choi;Kwang-Guk An
    • Korean Journal of Environmental Biology
    • /
    • v.41 no.2
    • /
    • pp.109-125
    • /
    • 2023
  • This study was conducted to select target fish species as baseline research for accumulation analysis of major hazardous chemicals entering the aquatic ecosystem in Korea and to analyze the impact on fish community. The test bed was selected from a sewage treatment plant, which could directly confirm the impact of the inflow of harmful chemicals, and the Geum River estuary where harmful chemicals introduced into the water system were concentrated. A multivariable metric model was developed to select target candidate fish species for hazardous chemical analysis. Details consisted of seven metrics: (1) commercially useful metric, (2) top-carnivorous species metric, (3) pollution fish indicator metric, (4) tolerance fish metric, (5) common abundant metric, (6) sampling availability (collectability) metric, and (7) widely distributed fish metric. Based on seven metric models for candidate fish species, eight species were selected as target candidates. The co-occurring dominant fish with target candidates was tolerant (50%), indicating that the highest abundance of tolerant species could be used as a water pollution indicator. A multi-metric fish-based model analysis for aquatic ecosystem health evaluation showed that the ecosystem health was diagnosed as "bad conditions". Physicochemical water quality variables also influenced fish feeding and tolerance guild in the testbed. Eight water quality parameters appeared high at the T1 site, indicating a large impact of discharging water from the sewage treatment plant. T2 site showed massive algal bloom, with chlorophyll concentration about 15 times higher compared to the reference site.