Ground truth and benchmarks for performance evaluation

Ayako Takeuchi; Michael Shneier; Tsai Hong Hong; Tommy Chang; Christopher Scrapper; Geraldine S. Cheok

doi:10.1117/12.485914

30 September 2003 Ground truth and benchmarks for performance evaluation

Ayako Takeuchi, Michael Shneier, Tsai Hong Hong, Tommy Chang, Christopher Scrapper, Geraldine S. Cheok

Proceedings Volume 5083, Unmanned Ground Vehicle Technology V; (2003) https://doi.org/10.1117/12.485914
Event: AeroSense 2003, 2003, Orlando, Florida, United States

Abstract

Progress in algorithm development and transfer of results to practical applications such as military robotics requires the setup of standard tasks, of standard qualitative and quantitative measurements for performance evaluation and validation. Although the evaluation and validation of algorithms have been discussed for over a decade, the research community still faces a lack of well-defined and standardized methodology. The range of fundamental problems include a lack of quantifiable measures of performance, a lack of data from state-of-the-art sensors in calibrated real-world environments, and a lack of facilities for conducting realistic experiments. In this research, we propose three methods for creating ground truth databases and benchmarks using multiple sensors. The databases and benchmarks will provide researchers with high quality data from suites of sensors operating in complex environments representing real problems of great relevance to the development of autonomous driving systems. At NIST, we have prototyped a High Mobility Multi-purpose Wheeled Vehicle (HMMWV) system with a suite of sensors including a Riegl ladar, GDRS ladar, stereo CCD, several color cameras, Global Position System (GPS), Inertial Navigation System (INS), pan/tilt encoders, and odometry . All sensors are calibrated with respect to each other in space and time. This allows a database of features and terrain elevation to be built. Ground truth for each sensor can then be extracted from the database. The main goal of this research is to provide ground truth databases for researchers and engineers to evaluate algorithms for effectiveness, efficiency, reliability, and robustness, thus advancing the development of algorithms.

Citation Download Citation

Ayako Takeuchi, Michael Shneier, Tsai Hong Hong, Tommy Chang, Christopher Scrapper, and Geraldine S. Cheok "Ground truth and benchmarks for performance evaluation", Proc. SPIE 5083, Unmanned Ground Vehicle Technology V, (30 September 2003); https://doi.org/10.1117/12.485914

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available