@article{ART002985296},
author={
홍승혜 and
신운섭 and
이규민 and
황동진 and
송상헌
},
title={Reasoning Ability of Deep Learning Models on Tautological Expressions in Korean},
journal={언어와 정보 사회},
issn={1598-1886},
year={2023},
volume={49},
pages={119-149},
doi={10.29211/soli.2023.49..005},
url={http://dx.doi.org/10.29211/soli.2023.49..005}
TY - JOUR
AU - 홍승혜
AU - 신운섭
AU - 이규민
AU - 황동진
AU - 송상헌
TI - Reasoning Ability of Deep Learning Models on Tautological Expressions in Korean
T2 - 언어와 정보 사회
PY - 2023
VL - 49
IS -
PB - 서강대학교 언어정보연구소
SP - 119-149
SN - 1598-1886
AB - The purpose of this paper is to evaluate the abductive reasoning capability of deep learning-based language models, with a specific focus on their ability to infer the meaning of tautological expressions. To achieve this, we conducted two experiments that required language models to understand the meaning of tautological expressions in different contexts. The first experiment was a binary classification task in which the models were presented with preceding sentences as contexts along with a tautological expression, and evaluated the adequacy of the tautological sentence. We employed three BERT-based models and assessed their capability based on the results of human evaluation. The second experiment involved having ChatGPT generate a coherent sentence that followed the preceding sentence and tautological expression. We categorized certain types of errors from the generated sentences that were deemed inappropriate. This study is significant in revealing the limitations of contemporary deep learning-based language models in terms of abductive reasoning and suggests a new task for standardized model evaluation.
KW - tautological expression, abductive reasoning, implicature, artificial intelligene, BERT, Chat-GPT
DO - 10.29211/soli.2023.49..005
UR - http://dx.doi.org/10.29211/soli.2023.49..005
ER -
홍승혜
,
신운섭
,
이규민
,
황동진
and
송상헌
(2023). Reasoning Ability of Deep Learning Models on Tautological Expressions in Korean. 언어와 정보 사회,
49,
119- 149.
홍승혜
,
신운섭
,
이규민
,
황동진
and
송상헌
. 2023,
“Reasoning Ability of Deep Learning Models on Tautological Expressions in Korean”,
언어와 정보 사회,
vol. 49,
pp. 119-149.
Available from: doi:10.29211/soli.2023.49..005
홍승혜
신운섭
et al.
이규민
황동진
송상헌
“Reasoning Ability of Deep Learning Models on Tautological Expressions in Korean” 언어와 정보 사회
49
pp. 119-149 (2023): 119.
홍승혜
,
신운섭
,
이규민
,
황동진
,
송상헌
. Reasoning Ability of Deep Learning Models on Tautological Expressions in Korean 언어와 정보 사회 [Internet]. 2023;
49
: 119-149.
Available from: doi:10.29211/soli.2023.49..005
홍승혜
,
신운섭
,
이규민
,
황동진
and
송상헌
. “Reasoning Ability of Deep Learning Models on Tautological Expressions in Korean” 언어와 정보 사회
49
(2023): 119-149. doi: 10.29211/soli.2023.49..005