Most of the existing methods for indoor point cloud semantic segmentation employ sophisticated structures to obtain local features, leading to high complexity. In addition, the current prevailing networks focus less on the extraction of global features of point clouds, which may damage the performance of the networks. Moreover, there are few networks that achieve the balance between accuracy and efficiency. To solve the above problems, we propose an improved network architecture based on anisotropic separable set abstraction network. It is more accurate and more efficient. First, we introduce an improved ASSA module to comprehensively consider the influence of the distance between the neighbor and the centroid on the neighborhood features, to better obtain the local features of the point clouds. Later, we design an inverse residual module, which optimizes the backbone network by scaling the receptive field to improve the extraction ability of the global features of the point clouds. Then we use a mixed pooling method to solve the disorder of point clouds and fuse both the coarse-grained and fine-grained features. Evaluated on the stanford large-scale 3D indoor spaces dataset, the experiment results illustrate that our method not only improves the extraction of global features and local features effectively, but also achieves an advanced performance among several mainstream point cloud semantic segmentation methods. Overall accuracy is 89.7%, mean class accuracy is 75.6%, mean intersection over union is 69.5%, and floating point operations are 15.9G.