Jongse Park
Emmanuel Amaro
Hadi Esmaeilzadeh
Skip Abstract Section
Abstract
Approximate computing trades quality of application output for higher efficiency and performance. Approximation is useful only if its impact on application output quality is acceptable to the users. However, there is a lack of systematic solutions and studies that explore users' perspective on the effects of approximation. In this paper, we seek to provide one such solution for the developers to probe and discover the boundary of quality loss that most users will deem acceptable. We propose AxGames, a crowdsourced solution that enables developers to readily infer a statistical common ground from the general public through three entertaining games. The users engage in these games by betting on their opinion about the quality loss of the final output while the AxGames framework collects statistics about their perceptions. The framework then statistically analyzes the results to determine the acceptable levels of quality for a pair of (application, approximation technique). The three games are designed such that they effectively capture quality requirements with various tradeoffs and contexts. To evaluate AxGames, we examine seven diverse applications that produce user perceptible outputs and cover a wide range of domains, including image processing, optical character recognition, speech to text conversion, and audio processing. We recruit 700 participants/users through Amazon's Mechanical Turk to play the games that collect statistics about their perception on different levels of quality. Subsequently, the AxGames framework uses the Clopper-Pearson exact method, which computes a binomial proportion confidence interval, to analyze the collected statistics for each level of quality. Using this analysis, AxGames can statistically project the quality level that satisfies a given percentage of users. The developers can use these statistical projections to tune the level of approximation based on the user experience. We find that the level of acceptable quality loss significantly varies across applications. For instance, to satisfy 90% of users, the level of acceptable quality loss is 2% for one application (image processing) and 26% for another (audio processing). Moreover, the pattern with which the crowd responds to approximation takes significantly different shape and form depending on the class of applications. These results confirm the necessity of solutions that systematically explore the effect of approximation on the end user experience.
- Hadi Esmaeilzadeh, Emily Blem, Renee St. Amant, Karthikeyan Sankaralingam, and Doug Burger. Dark silicon and the end of multicore scaling. In ISCA, 2011.Google Scholar
Digital Library
- Nikos Hardavellas, Michael Ferdman, Babak Falsafi, and Anastasia Ailamaki. Toward dark silicon in servers. IEEE Micro, 31 (4): 6--15, July--Aug. 2011.Google ScholarDigital Library
- Ganesh Venkatesh, Jack Sampson, Nathan Goulding, Saturnino Garcia, Vladyslav Bryksin, Jose Lugo-Martinez, Steven Swanson, and Michael Bedford Taylor. Conservation cores: Reducing the energy of mature computations. In ASPLOS, 2010.Google Scholar
- John Gantz and David Reinsel. Extracting value from chaos. http://www.emc.com/collateral/analyst-reports/idc-extracting-value-from-chaos-ar.pdf.Google Scholar
- Mehrzad Samadi, Janghaeng Lee, D. Anoushe Jamshidi, Amir Hormati, and Scott Mahlke. Sage: Self-tuning approximation for graphics engines. In MICRO, 2013.Google Scholar
- Mehrzad Samadi, Davoud Jamshidi, Janghaeng Lee, and Scott Mahlke. Paraprox: Pattern-based approximation for data parallel applications. In ASPLOS, 2014.Google ScholarDigital Library
- Esmaeilzadeh, Sampson, Ceze, and Burger]truffleHadi Esmaeilzadeh, Adrian Sampson, Luis Ceze, and Doug Burger. Architecture support for disciplined approximate programming. In ASPLOS, 2012.Google ScholarDigital Library
- Hadi Esmaeilzadeh, Adrian Sampson, Luis Ceze, and Doug Burger. Neural acceleration for general-purpose approximate programs. In MICRO, 2012.Google ScholarDigital Library
- Renée St. Amant, Amir Yazdanbakhsh, Jongse Park, Bradley Thwaites, Hadi Esmaeilzadeh, Arjang Hassibi, Luis Ceze, and Doug Burger. General-purpose code acceleration with limited-precision analog computation. In ISCA, 2014.Google Scholar
- Adrian Sampson, Werner Dietl, Emily Fortuna, Danushen Gnanapragasam, Luis Ceze, and Dan Grossman. EnerJ: Approximate data types for safe and general low-power computation. In PLDI, 2011.Google ScholarDigital Library
- Jongse Park, Hadi Esmaeilzadeh, Xin Zhang, Mayur Naik, and William Harris. Flexjava: Language support for safe and modular approximate programming. In FSE, 2015.Google ScholarDigital Library
- Stelios Sidiroglou, Sasa Misailovic, Henry Hoffmann, and Martin Rinard. Managing performance vs. accuracy trade-offs with loop perforation. In FSE, 2011.Google ScholarDigital Library
- Michael Carbin, Sasa Misailovic, and Martin Rinard. Verifying quantitative reliability for programs that execute on unreliable hardware. In OOPSLA, 2013.Google ScholarDigital Library
- Inigo Goiri, Ricardo Bianchini, Santosh Nagarakatte, and Thu D. Nguyen. Approxhadoop: Bringing approximations to mapreduce frameworks. In ASPLOS, 2015.Google ScholarDigital Library
- Joshua San Miguel, Mario Badr, and Natalie Enright Jerger. Load value approximation. In MICRO, 2014.Google ScholarDigital Library
- Adrian Sampson, Jacob Nelson, Karin Strauss, and Luis Ceze. Approximate storage in solid-state memories. In MICRO, 2013.Google ScholarDigital Library
- C. J. Clopper and E. S. Pearson. The use of confidence or fiducial limits illustrated in the case of the binomial. Biometrika, 26 (4): 404--413, Dec. 1934.Google ScholarCross Ref
- Sean Wallis. Binomial confidence intervals and contingency tests: mathematical fundamentals and the evaluation of alternative methods. Journal of Quantitative Linguistics, 20 (3): 178--208, July. 2013.Google ScholarCross Ref
- A/b testing. https://en.wikipedia.org/wiki/A/B_testing.Google Scholar
- Edwin B. Wilson. Probable inference, the law of succession, and statistical inference. Journal of the American Statistical Association, 22 (158): 209--212, Jun. 1927.Google ScholarCross Ref
- Jeff Sauro and James R. Lewis. Estimating completion rates from small samples using binomial confidence intervals: Comparisons and recommendations. In HFES, 2005.Google ScholarCross Ref
- Morris H. DeGroot. Probability and Statistics. Chapman & Hall, 1974.Google Scholar
- Lame mp3 encoder. http://lame.sourceforge.net.Google Scholar
- Speech recognition toolkit. http://cmusphinx.sourceforge.net.Google Scholar
- John W Ratcliff and David E Metzener. Pattern matching: The gestalt approach. Dr. Dobb's Journal, 13 (7): 46, 1988.Google Scholar
- Amir Yazdanbakhsh, Jongse Park, Hardik Sharma, Pejman Lofti-Kamran, and Hadi Esmaeilzadeh. Neural acceleration for gpu throughput processors. In MICRO, 2015.Google ScholarDigital Library
- Amir Yazdanbakhsh, Divya Mahajan, Bradley Thwaites, Jongse Park, Anandhavel Nagendrakumar, Sindhuja Sethuraman, Kartik Ramkrishnan, Nishanthi Ravindran, Rudra Jariwala, Abbas Rahimi, Hadi Esmaeilzadeh, and Kia Bazargan. Axilog: Language support for approximate hardware design. In DATE, 2015.Google ScholarDigital Library
- Sasa Misailovic, Michael Carbin, Sara Achour, Zichao Qi, and Martin Rinard. Chisel: Reliability- and accuracy-aware optimization of approximate computational kernels. In OOPSLA, 2014.Google ScholarDigital Library
- Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009. URL http://image-net.org.Google ScholarCross Ref
- Freesound.org. https://freesound.org.Google Scholar
- Voxforge. https://developer.nvidia.com/cuda-llvm-compiler.Google Scholar
- The new york times timesmachine. http://timesmachine.nytimes.com.Google Scholar
- Amazon's mechanical turk. https://www.mturk.com.Google Scholar
- Avadh Patel, Furat Afram, Shunfei Chen, and Kanad Ghose. MARSSx86: A full system simulator for x86 CPUs. In DAC, 2011.Google Scholar
- Sheng Li, Jung Ho Ahn, Richard D. Strong, Jay B. Brockman, Dean M. Tullsen, and Norman P. Jouppi. McPAT: An integrated power, area, and timing modeling framework for multicore and manycore architectures. In MICRO, 2009.Google ScholarDigital Library
- Jaegul Choo, Changhyun Lee, Hannah Kim, Hanseung Lee, Barry L. Drake, and Haesun Park. Apolo: Making sense of large network data by combining rich user interaction and machine learning. In VAST, 2014.Google Scholar
- Duen Horng (Polo) Chau, Aniket Kittur, Jason I. Hong, and Christos Faloutsos. Apolo: Making sense of large network data by combining rich user interaction and machine learning. In CHI, 2011.Google ScholarDigital Library
- Song Liu, Karthik Pattabiraman, Thomas Moscibroda, and Benjamin Zorn. Flikker: Saving refresh-power in mobile devices through critical data partitioning. In ASPLOS, 2011.Google ScholarDigital Library
- Sampson, Nelson, Strauss, and Ceze]flashAdrian Sampson, Jacob Nelson, Karin Strauss, and Luis Ceze. Approximate storage in solid-state memories. In MICRO, 2013.Google ScholarDigital Library
- Lakshmi N. Chakrapani, Bilge E. S. Akgul, Suresh Cheemalavagu, Pinar Korkmaz, Krishna V. Palem, and Balasubramanian Seshasayee. Ultra-efficient (embedded) SOC architectures based on probabilistic CMOS (PCMOS) technology. In DATE, 2006.Google ScholarDigital Library
- Sriram Narayanan, John Sartori, Rakesh Kumar, and Douglas Jones. Scalable stochastic processors. In DATE, 2010.Google ScholarDigital Library
- Rajamohana Hegde and Naresh Shanbhag. Energy-efficient signal processing via algorithmic noise-tolerance. In ISLPED, 1999.Google ScholarDigital Library
- Larkhoon Leem, Hyungmin Cho, Jason Bau, Quinn Jacobson, and Subhasish Mitra. ERSA: error resilient system architecture for probabilistic applications. In DATE, 2010.Google ScholarDigital Library
- Sasa Misailovic, Stelios Sidiroglou, Hank Hoffman, and Martin Rinard. Quality of service profiling. In ICSE, 2010.Google ScholarDigital Library
- Martin Rinard, Henry Hoffmann, Sasa Misailovic, and Stelios Sidiroglou. Patterns and statistical analysis for understanding reduced resource computing. In Onward!, 2010.Google Scholar
- Woongki Baek and Trishul Chilimbi. Green: a framework for supporting energy-conscious programming using controlled approximation. In PLDI, 2010.Google Scholar
- Jason Ansel, Cy Chan, Yee Lok Wong, Marek Olszewski, Qin Zhao, Alan Edelman, and Saman Amarasinghe. Petabricks: a language and compiler for algorithmic choice. In PLDI, 2009.Google ScholarDigital Library
- John Sartori and Rakesh Kumar. Branch and data herding: Reducing control and memory divergence for error-tolerant gpu applications. IEEE Transactions on Multimedia, 15 (2), 2013.Google ScholarDigital Library
- Carlos Alvarez, Jesus Corbal, and Mateo Valero. Fuzzy memoization for floating-point multimedia applications. IEEE Trans. on Computers, 54 (7), 2005.Google ScholarDigital Library
- Jose-Maria Arnau, Joan-Manuel Parcerisa, and Polychronis Xekalakis. Eliminating redundant fragment shader executions on a mobile gpu via hardware memoization. In ISCA, 2014.Google ScholarDigital Library
- Marc de Kruijf, Shuou Nomura, and Karthikeyan Sankaralingam. Relax: an architectural framework for software recovery of hardware faults. In ISCA, 2010.Google Scholar
- Xuanhua Li and Donald Yeung. Application-level correctness and its impact on fault tolerance. In HPCA, 2007.Google ScholarDigital Library
- Xuanhua Li and Donald Yeung. Exploiting application-level correctness for low-cost fault tolerance. Journal of Instruction-Level Parallelism, 2008.Google Scholar
- Marc de Kruijf and Karthikeyan Sankaralingam. Exploring the synergy of emerging workloads and silicon reliability trends. In SELSE, 2009.Google Scholar
- Yuntan Fang, Huawei Li, and Xiaowei Li. A fault criticality evaluation framework of digital systems for error tolerant video applications. In ATS, 2011.Google ScholarDigital Library
- Vicky Wong and Mark Horowitz. Soft error resilience of probabilistic inference applications. In SELSE, 2006.Google Scholar
- Swagath Venkataramani, Vinay K. Chippa, Srimat T. Chakradhar, Kaushik Roy, and Anand Raghunathan. Quality programmable vector processors for approximate computing. In MICRO, 2013.Google ScholarDigital Library
- Ashish Ranjan, Arnab Raha, Swagath Venkataramani, Kaushik Roy, and Anand Raghunathan. Aslan: Synthesis of approximate sequential circuits. In DATE, 2014.Google ScholarCross Ref
- Swagath Venkataramani, Amit Sabne, Vivek Kozhikkottu, Kaushik Roy, and Anand Raghunathan. Salsa: Systematic logic synthesis of approximate circuits. In DAC, 2012.Google ScholarDigital Library
- Jin Miao, A. Gerstlauer, and M. Orshansky. Approximate logic synthesis under general error magnitude and frequency constraints. In ICCAD, 2013.Google ScholarDigital Library
- Kumud Nepal, Yueting Li, R. Iris Bahar, and Sherief Reda. ABACUS: A technique for automated behavioral synthesis of approximate computing circuits. In DATE, 2014.Google ScholarDigital Library
- Avinash Lingamneni, Christian Enz, Krishna Palem, and Christian Piguet. Synthesizing parsimonious inexact circuits through probabilistic design techniques. ACM Transactions on Embedded Computing Systems, 12 (2s): 93:1--93:26, May 2013.Google ScholarDigital Library
- Avinash Lingamneni, Kirthi Krishna Muntimadugu, Christian Enz, Richard M. Karp, Krishna V. Palem, and Christian Piguet. Algorithmic methodologies for ultra-efficient inexact architectures for sustaining technology scaling. In CF, 2012.Google Scholar
- Luis Von Ahn, Benjamin Maurer, Colin McMillen, David Abraham, and Manuel Blum. recaptcha: Human-based character recognition via web security measures. Science, 321 (5895): 1465--1468, 2008.Google ScholarCross Ref
- Daniel W Barowy, Charlie Curtsinger, Emery D Berger, and Andrew McGregor. Automan: A platform for integrating human-based and digital computation. In OOPSLA, 2012.Google ScholarDigital Library
- Greg Little, Lydia B Chilton, Max Goldman, and Robert C Miller. Turkit: human computation algorithms on mechanical turk. In UIST, 2010.Google Scholar
- Bryan C Russell, Antonio Torralba, Kevin P Murphy, and William T Freeman. Labelme: a database and web-based tool for image annotation. International Journal of Computer Vision, 77 (1--3): 157--173, 2008.Google ScholarDigital Library
- Luis Von Ahn. Games with a purpose. Computer, 39 (6): 92--94, 2006.Google ScholarDigital Library
- Luis Von Ahn, Ruoran Liu, and Manuel Blum. Peekaboom: a game for locating objects in images. In CHI, 2006.Google Scholar
- Luis Von Ahn and Laura Dabbish. Labeling images with a computer game. In CHI, 2004.Google ScholarDigital Library
- Werner Dietl, Stephanie Dietzel, Michael D Ernst, Nathaniel Mote, Brian Walker, Seth Cooper, Timothy Pavlik, and Zoran Popović. Verification games: Making verification fun. In FTfJP, 2012.Google ScholarDigital Library
- Seth Cooper, Firas Khatib, Adrien Treuille, Janos Barbero, Jeehyung Lee, Michael Beenen, Andrew Leaver-Fay, David Baker, Zoran Popović, and Foldit players. Predicting protein structures with a multiplayer online game. Nature, 466 (7307): 756--760, 2010.Google ScholarCross Ref
- Henry Hoffmann, Stelios Sidiroglou, Michael Carbin, Sasa Misailovic, Anant Agarwal, and Martin Rinard. Dynamic knobs for responsive power-aware computing. In ASPLOS, 2011.Google ScholarDigital Library
- J. Bornholt, T. Mytkowicz, and K. McKinley. Uncertain <T>: A first-order type for uncertain data. In ASPLOS, 2014.Google ScholarDigital Library
- Michael Ringenburg, Adrian Sampson, Isaac Ackerman, Luis Ceze, and Dan Grossman. Monitoring and debugging the quality of results in approximate programs. In ASPLOS, 2015.Google ScholarDigital Library
- Phillip Stanley-Marbell and Martin Rinard. Lax: Driver interfaces for approximate sensor device access. In HotOS, 2015.Google Scholar
Index Terms
- AxGames: Towards Crowdsourcing Quality Target Determination in Approximate Computing
Recommendations
AxGames: Towards Crowdsourcing Quality Target Determination in Approximate Computing
ASPLOS '16: Proceedings of the Twenty-First International Conference on Architectural Support for Programming Languages and Operating SystemsApproximate computing trades quality of application output for higher efficiency and performance. Approximation is useful only if its impact on application output quality is acceptable to the users. However, there is a lack of systematic solutions and ...
AxGames: Towards Crowdsourcing Quality Target Determination in Approximate Computing
ASPLOS'16Approximate computing trades quality of application output for higher efficiency and performance. Approximation is useful only if its impact on application output quality is acceptable to the users. However, there is a lack of systematic solutions and ...
A Glimpse Far into the Future: Understanding Long-term Crowd Worker Quality
CSCW '17: Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social ComputingMicrotask crowdsourcing is increasingly critical to the creation of extremely large datasets. As a result, crowd workers spend weeks or months repeating the exact same tasks, making it necessary to understand their behavior over these long periods of ...
Comments