Color errors in scanners arise from two sources: the non-colorimetric nature of the scanner sensitivities and the measurement noise. Several measures of goodness have been used to evaluate scanners based on these errors. In this paper, the trustworthiness of these measures is studied through simulations. A new measure incorporating both the above sources of errors and providing excellent agreement with perceived color error is also presented.