Zubair Qazi
ABSTRACT
As natural language models like ChatGPT become increasingly prevalent in applications and services, the need for robust and accurate methods to detect their output is of paramount importance. In this paper, we present GPT Reddit Dataset (GRiD), a novel Generative Pretrained Transformer (GPT)-generated text detection dataset designed to assess the performance of detection models in identifying generated responses from ChatGPT. The dataset consists of a diverse collection of context-prompt pairs based on Reddit, with human-generated and ChatGPT-generated responses. We provide an analysis of the dataset's characteristics, including linguistic diversity, context complexity, and response quality. To showcase the dataset's utility, we benchmark several detection methods on it, demonstrating their efficacy in distinguishing between human and ChatGPT-generated responses. This dataset serves as a resource for evaluating and advancing detection techniques in the context of ChatGPT and contributes to the ongoing efforts to ensure responsible and trustworthy AI-driven communication on the internet. Finally, we propose GpTen, a novel tensor-based GPT text detection method that is semi-supervised in nature since it only has access to human-generated text and performs on par with fully-supervised baselines.
Supplemental Material
- Tom B. Brown et al. 2020. Language Models are Few-Shot Learners. CoRR , Vol. abs/2005.14165 (2020). showeprint[arXiv]2005.14165Google Scholar
- Evan Crothers, Nathalie Japkowicz, and Herna Viktor. 2023. Machine Generated Text: A Comprehensive Survey of Threat Models and Detection Methods. arxiv: 2210.07321 [cs.CL]Google Scholar
- Gisel Bastidas Guacho, Sara Abdali, Neil Shah, and Evangelos E. Papalexakis. 2018. Semi-supervised Content-based Detection of Misinformation via Tensor Embeddings. arxiv: 1804.09088 [cs.LG]Google Scholar
- Zied Haj-Yahia, Adrien Sieg, and Léa A Deleris. 2019. Towards unsupervised text classification leveraging experts and word embeddings. In Proceedings of the 57th annual meeting of the Association for Computational Linguistics. 371--379.Google ScholarCross Ref
- Karen Sparck Jones. 1972. A statistical interpretation of term specificity and its application in retrieval. Journal of documentation, Vol. 28, 1 (1972), 11--21.Google ScholarCross Ref
- Qian Li, Hao Peng, Jianxin Li, Congying Xia, Renyu Yang, Lichao Sun, Philip S. Yu, and Lifang He. 2020. A Survey on Text Classification: From Shallow to Deep Learning. CoRR , Vol. abs/2008.00364 (2020). showeprint[arXiv]2008.00364Google Scholar
- Sebastian Ruder, Matthew E Peters, Swabha Swayamdipta, and Thomas Wolf. 2019. Transfer learning in natural language processing. In NAACL: Tutorials. 15--18.Google Scholar
- Yue Zhao, Zain Nasrullah, and Zheng Li. 2019b. PyOD: A Python Toolbox for Scalable Outlier Detection. Journal of Machine Learning Research, Vol. 20, 96 (2019), 1--7. http://jmlr.org/papers/v20/19-011.htmlGoogle Scholar
- Zhenjie Zhao, Andrew Cattle, Evangelos Papalexakis, and Xiaojuan Ma. 2019a. Embedding lexical features via tensor decomposition for small sample humor recognition. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).Google ScholarCross Ref
Index Terms
- GPT-generated Text Detection: Benchmark Dataset and Tensor-based Detection Method
Recommendations
Tensor-based anomaly detection
Traditional spectral-based methods such as PCA are popular for anomaly detection in a variety of problems and domains. However, if data includes tensor (multiway) structure (e.g. space-time-measurements), some meaningful anomalies may remain invisible ...
Outlier/Anomaly Detection of Univariate Time Series: A Dataset Collection and Benchmark
Big Data Analytics and Knowledge DiscoveryAbstractIn this paper, we present an extensive collection of outlier/anomaly detection tasks to identify unusual series from a given time series dataset. The presented work is based on the popular UCR time series classification archive. In addition to the ...
A unified benchmark for the unknown detection capability of deep neural networks
AbstractDeep neural networks have achieved outstanding performance over various tasks, but they have a critical issue: over-confident predictions even for completely unknown samples. Many studies have been proposed to successfully filter out ...
Highlights- An unknown detection task is proposed to evaluate the detection capability of NNs.
Comments