Photogrammetry has been considered as the most efficient technique for extracting 3D spatial information in the last few decades. However, the point-by-point measurement of using floating mark has become the bottleneck while reconstructing the 3D city model. In this paper, we expanded the floating mark to the floating model based on the concept of model-based building extraction. The measuring tool is no longer an abstract point but many kinds of 3D model, which can be scaled, rotated, or moved in the space. The floating model is defined with a datum point indicates the model's position. Each kind of models is associated with a set of pose parameters to describe its rotation about the three orthogonal axes and shape parameters to describe its scales along predefined directions. In other words, the floating model is a flexible entity floating in the space, and can be adjusted to fit the object by these parameters. If the model parameters are good enough to represent the 3D spatial information of the object, the projection of the floating model on every overlapped image will all be coincident to the object's outlines. In order to meet the needs of building reconstruction, we designed several primitive models that can be categorized into four kinds: point, line segment, surface, and solid. By means of human-computer interface, the operator is able to choose the most suitable model and measure the object on multiple images. The complicated building can be split into several parts, modeled part-by-part, and finally joined into one complex model. Based on the floating model theory and model-based building extraction, we proposed a semi-automated strategy. A friendly human-machine interface is designed for the operator to choose and adjust the floating model to fit the images manually. Then, the computer calculate the optimal fit by an ad hoc Least-Squares Model-Image Fitting algorithm. Thus the 3D spatial information can be extracted object-by-object, which increases the efficiency and accuracy.