Motion tracking with optical devices such as surveillance cameras can be particularly powerful for measuring earthquake induced motions. We propose a vision-based system with motion tracking for providing additional safety functions to detect seismic damage to interior elements and thus reduce risks in a building. Shaking table tests were carried out to evaluate the system's ability to detect structural collapse and overturning of interior elements using conventional video cameras. Results from this exploratory study show that vision-based measurements are promising in terms of capturing small to large deformations and identifying various kinds of seismic damage.