In this paper we present a neural network (NN) architecture for abnormal events detection in a surveillance system. When training such system, it is a challenge that only normal samples are available for training. In addition, there are various contents inside a surveillance video frame, such like lawn, pedestrian walking area, trees, etc., and each has its own “typical/atypical” pattern. In solving the first problem, we propose a scheme to separate training data into “typical” and “atypical” that serve the roles of negative and positive training samples. In the second problem, we first divide a frame into several blocks. Then, for each block, an adaptive cascaded 3D-CNN classifier is trained. In this way, most of irrelevant blocks (normal) are discarded and only those abnormal or confusing normal blocks are kept for further computations. We evaluate our approach on popular dataset and show that our approach is competitive to state-of-the-art methods.