This research proposes an efficient strip steel surface defect classification model (ASNet) based on convolutional neural network (CNN), which can run in real time on commonly used serial computing platforms. We only used a very shallow CNN structure to extract features of the defect images, and an attention layer which makes the model ignore some irrelevant noise and obtain an effective description of the defects is designed. In addition, a nonlinear perceptron is added to the top of the model to recognize defects based on the extracted features. On the strip steel surface defect image dataset NEU-CLS, our model achieves an average classification accuracy of 99.9%, while the number of parameters of the model is only 0.041 M and the computational complexity of the model is 98.1 M FLOPs. It can meet the requirements of real-time operation and large-scale deployment on a common serial computing platform with high recognition accuracy.