The conversion and loss of primary rainforest in Southeast Asia is presumed to affect many animal assemblages in terms of their diversity and species composition. We studied the responses of non-volant small mammals to forest loss and degradation caused by logging and the establishment of large-scale oil palm plantations located in the north-eastern part of Borneo in Southeast Asia. We found that habitat types (forest versus plantation) were important determinant of species occurrences and assemblage compositions. However, within forest habitats irrespective of whether logged or unlogged, similarities in assemblage compositions of sites close to each other underline the importance of geographic distance in shaping small mammal assemblages. In terms of conservation management, it is imperative to incorporate the size and overall surrounding environment of a forest into forest management concepts, and although not equivalent to areas of primary forest, old regenerating secondary forest needs to be considered as an important component for the preservation of small mammal species diversity. In contrast, oil palm plantations are clearly of little or no importance for the conservation of the non-volant small mammal fauna and this habitat may also act as an effective barrier to the dispersal of the small mammals.