The model drift problem is an inevitable problem for online visual tracking. Model drift can amplify the false tracking result over time and lead to failed tracking finally. Background-aware correlation filter (BACF) tracker obtains the training samples from negative background patches instead of shifted foreground patches and can mitigate the drifting to some extent. However, in complex scenes, such as occlusion, deformation, the inappropriate update of BACF may lead to model drift. We propose an anti-drift background-aware correlation filter via introducing the temporal consistency constraint into BACF. That is, we combine the excellent ability of BACF to distinguish the foreground and background with the ability of the temporal consistency constraint to stabilize model changes, and improve anti-drift performance of BACF. To improve computation efficient, we offer a fast algorithm via alternative Direction Method of Multipliers (ADMM) in the frequency domain. Besides, we design a simple yet effective adaptive feature channel selection method, which can further improve the success rate and precision of the tracker in complex scenes. Our proposed tracker with hand-crafted features achieves a gain of 2.2%, 3.3%, and 5.3% in AUC success rate on OTB-2013, OTB-2015, and Temple Color-128 dataset, respectively. Our proposed tracker with deep features achieves AUC success rate of 69.1% on OTB-2013 dataset. Besides, our proposed tracker with hand-crafted features achieves a gain of 3.94% in EAO on VOT-2018 dataset. Moreover, the proposed tracker performs favorably against several representative state-of-the-art methods regarding precision and success rate.