Surface-enhanced Raman spectroscopy (SERS) has shown promising potential in cancer screening. In practical applications, Raman spectra are often affected by deviations from the spectrometer, changes in measurement environments, and anomalies in spectrum characteristic peak intensities due to improper sample storage. Previous research has overlooked the presence of outliers in categorical data, leading to significant impacts on model learning outcomes. In this study, we propose a novel method, called Principal Component Analysis and Density Based Spatial Clustering of Applications with Noise (PCA-DBSCAN) to effectively remove outliers. This method employs dimensionality reduction and spectral data clustering to identify and remove outliers. The PCA-DBSCAN method introduces adjustable parameters (Eps and MinPts) to control the clustering effect. The effectiveness of the proposed PCA-DBSCAN method is verified through modeling on outlier-removed datasets. Further refinement of the machine learning model and PCA-DBSCAN parameters resulted in the best cancer screening model, achieving 97.41% macro-average recall and 97.74% macro-average F1-score. This paper introduces a new outlier removal method that significantly improves the performance of the SERS cancer screening model. Moreover, the proposed method serves as inspiration for outlier detection in other fields, such as biomedical research, environmental monitoring, manufacturing, quality control, and hazard prediction.