We consider the quantum and classical noise limits to position measurement and force detection by an atomic force microscope (AFM) with an optical readout of cantilever position. We model this by treating the cantilever as a perfectly reflecting mirror for a highly damped optical cavity. There are three sources of noise: the shot noise in the output laser measurement, the thermal noise in the cantilever, and the measurement back-action noise. This last source of noise becomes large for good measurements, measurements for which there is a high correlation between the output phase of the light and the changing position of the cantilever. The back action simultaneously drives a diffusion process in momentum and diagonalizes the cantilever state in the position basis. This latter result is ‘‘state reduction.’’ Explicit expressions for the rate of the measurement back action in terms of the device parameters are given. We also calculate the signal-to-noise ratio in the limit of a bad cavity. A comparison to recent experiments suggests that current AFMs are not quantum limited.

• Received 27 April 1994

DOI:https://doi.org/10.1103/PhysRevA.50.5256