Sojeong Yun
Youn-Kyung Lim
ABSTRACT
Sensors and actuators are crucial components of a do-it-yourself (DIY) smart home system that enables users to construct smart home features successfully. In addition, machine learning (ML) (e.g., ML-intensive camera sensors) can be applied to sensor technology to increase its accuracy. Although camera sensors are often utilized in homes, research on user experiences with DIY smart home systems employing camera sensors is still in its infancy. This research investigates novel user experiences while constructing DIY smart home features using an ML-intensive camera sensor in contrast to commonly used IoT sensors. Thus, we conducted a seven-day field diary study with 12 families who were given a DIY smart home kit. Here, we assess the five characteristics of the camera sensor as well as the potential and challenges of utilizing the camera sensor in the DIY smart home and discuss the opportunities to address existing DIY smart home issues.
Supplemental Material
- R Aiswarya, Vaishali Anil, Jayasree Raveendran, Gijs Van Wijk, and Shantanu Talukdar. 2021. Living as a Service (LaaS): An Experiment to Evaluate Living Experience in Connected Homes in the Netherlands.. In CHI Extended Abstracts. 360–1.Google Scholar
- Virginia Braun and Victoria Clarke. 2006. Using thematic analysis in psychology. Qualitative research in psychology 3, 2 (2006), 77–101.Google Scholar
- Nathalie Bressa, Jo Vermeulen, and Wesley Willett. 2022. Data Every Day: Designing and Living with Personal Situated Visualizations. In CHI Conference on Human Factors in Computing Systems. 1–18.Google ScholarDigital Library
- Will Burns. 2018. The “Nestie Awards” From Nest Is A Content Idea Firing On All Cylinders. Forbes.Google Scholar
- Yu-Ting Cheng, Mathias Funk, Wenn-Chieh Tsai, and Lin-Lin Chen. 2019. Peekaboo Cam: Designing an Observational Camera for Home Ecologies Concerning Privacy. In Proceedings of the 2019 on Designing Interactive Systems Conference. 823–836.Google ScholarDigital Library
- Yi-Shyuan Chiang, Ruei-Che Chang, Yi-Lin Chuang, Shih-Ya Chou, Hao-Ping Lee, I-Ju Lin, Jian-Hua Jiang Chen, and Yung-Ju Chang. 2020. Exploring the design space of user-system communication for smart-home routine assistants. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1–14.Google ScholarDigital Library
- Jun-Ho Choi, Kyungmin Kim, Taejin Park, Junho Yun, Jong-Hwan Lee, Songkuk Kim, Hyunjung Shim, and Jong-Seok Lee. 2020. Real-time integrated human activity recognition system based on multimodal user understanding. In Proceedings of the 25th International Conference on Intelligent User Interfaces Companion. 89–90.Google ScholarDigital Library
- Jun-Ho Choi and Jong-Seok Lee. 2018. Confidence-based deep multimodal fusion for activity recognition. In Proceedings of the 2018 ACM International Joint Conference and 2018 International Symposium on Pervasive and Ubiquitous Computing and Wearable Computers. 1548–1556.Google ScholarDigital Library
- Cubriod. 2022. Cubriod coding blocks. https://www.cubroid.co.kr/shopping/?idx=1Google Scholar
- Cubroid. 2022. Curo AI camera. https://www.cubroid.co.kr/shopping/?idx=78Google Scholar
- Audrey Desjardins, Heidi R Biggs, Cayla Key, and Jeremy E Viny. 2020. IoT data in the home: Observing entanglements and drawing new encounters. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1–13.Google ScholarDigital Library
- Audrey Desjardins, Jeremy E Viny, Cayla Key, and Nouela Johnston. 2019. Alternative avenues for IoT: Designing with non-stereotypical homes. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 1–13.Google ScholarDigital Library
- Audrey Desjardins and Ron Wakkary. 2016. Living in a prototype: A reconfigured space. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. 5274–5285.Google ScholarDigital Library
- Dan Ding, Rory A Cooper, Paul F Pasquina, and Lavinia Fici-Pasquina. 2011. Sensor technology for smart homes. Maturitas 69, 2 (2011), 131–136.Google ScholarCross Ref
- Yuanyuan Feng, Yaxing Yao, and Norman Sadeh. 2021. A design space for privacy choices: Towards meaningful privacy control in the internet of things. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1–16.Google ScholarDigital Library
- Micro:bit Educational Foundation. 2022. Micro:bit sensor. https://microbit.org/ko/Google Scholar
- Carolina Fuentes, Martin Porcheron, Joel E Fischer, Enrico Costanza, Obaid Malilk, and Sarvapali D Ramchurn. 2019. Tracking the consumption of home essentials. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 1–13.Google ScholarDigital Library
- Carolina Fuentes, Martin Porcheron, Joel E Fischer, Enrico Costanza, Nervo Verdezoto, Valeria Herskovic, Oren Zuckerman, and Leila Takayama. 2019. New directions for the IoT: automate, share, build, and care. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems. 1–8.Google ScholarDigital Library
- Christine Geeng and Franziska Roesner. 2019. Who’s in control? Interactions in multi-user smart homes. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 1–13.Google ScholarDigital Library
- Taesik Gong, Hyunsung Cho, Bowon Lee, and Sung-Ju Lee. 2019. Knocker: Vibroacoustic-based object recognition with smartphones. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 3, 3 (2019), 1–21.Google ScholarDigital Library
- Google. 2022. Google’s Teachable Machine. https://teachablemachine.withgoogle.com/Google Scholar
- Anhong Guo, Anuraag Jain, Shomiron Ghose, Gierad Laput, Chris Harrison, and Jeffrey P Bigham. 2018. Crowd-ai camera sensing in the real world. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 2, 3 (2018), 1–20.Google ScholarDigital Library
- Neilly H. Tan, Brian Kinnee, Dana Langseth, Sean A. Munson, and Audrey Desjardins. 2022. Critical-Playful Speculations with Cameras in the Home. In CHI Conference on Human Factors in Computing Systems. 1–22.Google Scholar
- Neilly H. Tan, Richmond Y. Wong, Audrey Desjardins, Sean A. Munson, and James Pierce. 2022. Monitoring Pets, Deterring Intruders, and Casually Spying on Neighbors: Everyday Uses of Smart Home Cameras. In CHI Conference on Human Factors in Computing Systems. 1–25.Google ScholarDigital Library
- Xin Hong, Chris Nugent, Maurice Mulvenna, Sally McClean, Bryan Scotney, and Steven Devlin. 2009. Evidential fusion of sensor data for activity recognition in smart homes. Pervasive and Mobile Computing 5, 3 (2009), 236–252.Google ScholarDigital Library
- Amy S Hwang and Jesse Hoey. 2012. Smart Home, The Next Generation: Closing the Gap between Users and Technology.. In AAAI Fall Symposium: Artificial Intelligence for Gerontechnology.Google Scholar
- Timo Jakobi, Gunnar Stevens, Nico Castelli, Corinna Ogonowski, Florian Schaub, Nils Vindice, Dave Randall, Peter Tolmie, and Volker Wulf. 2018. Evolving needs in IoT control and accountability: A longitudinal study on smart home intelligibility. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 2, 4 (2018), 1–28.Google ScholarDigital Library
- Yeseong Kim, Mohsen Imani, and Tajana S Rosing. 2018. Efficient human activity recognition using hyperdimensional computing. In Proceedings of the 8th International Conference on the Internet of Things. 1–6.Google ScholarDigital Library
- Vinay Koshy, Joon Sung Sung Park, Ti-Chung Cheng, and Karrie Karahalios. 2021. “We Just Use What They Give Us”: Understanding Passenger User Perspectives in Smart Homes. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1–14.Google ScholarDigital Library
- Martin J Kraemer, Ulrik Lyngs, Helena Webb, and Ivan Flechais. 2020. Further exploring communal technology use in smart homes: social expectations. In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems. 1–7.Google ScholarDigital Library
- Albrecht Kurze, Andreas Bischof, Sören Totzauer, Michael Storz, Maximilian Eibl, Margot Brereton, and Arne Berger. 2020. Guess the data: Data work to understand how people make sense of and use simple sensor data from homes. In Proceedings of the 2020 CHI Conference on Human Factors in Computing systems. 1–12.Google ScholarDigital Library
- Gierad Laput, Karan Ahuja, Mayank Goel, and Chris Harrison. 2018. Ubicoustics: Plug-and-play acoustic activity recognition. In Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology. 213–224.Google ScholarDigital Library
- Gierad Laput, Walter S Lasecki, Jason Wiese, Robert Xiao, Jeffrey P Bigham, and Chris Harrison. 2015. Zensors: Adaptive, rapidly deployable, human-intelligent sensor feeds. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. 1935–1944.Google ScholarDigital Library
- Gierad Laput, Yang Zhang, and Chris Harrison. 2017. Synthetic sensors: Towards general-purpose sensing. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. 3986–3999.Google ScholarDigital Library
- Kyung-Ryong Lee, Somi Ju, Temirlan Dzhoroev, Geonil Goh, Moon-Hwan Lee, and Young-Woo Park. 2020. DayClo: an everyday table clock providing interaction with personal schedule data for self-reflection. In Proceedings of the 2020 ACM Designing Interactive Systems Conference. 1793–1806.Google ScholarDigital Library
- Dawei Liang, Wenting Song, and Edison Thomaz. 2020. Characterizing the Effect of Audio Degradation on Privacy Perception And Inference Performance in Audio-Based Human Activity Recognition. In 22nd International Conference on Human-Computer Interaction with Mobile Devices and Services. 1–10.Google Scholar
- Changmo Nam, Jun-Cheol Park, and Dae-Shik Kim. 2016. Indoor Human Activity Recognition with Contextual Cues in Videos. In Proceedings of the 10th International Conference on Ubiquitous Information Management and Communication. 1–4.Google ScholarDigital Library
- Mahda Noura, Sebastian Heil, and Martin Gaedke. 2020. Natural language goal understanding for smart home environments. In Proceedings of the 10th International Conference on the Internet of Things. 1–8.Google ScholarDigital Library
- Erick Oduor, Carman Neustaedter, William Odom, Anthony Tang, Niala Moallem, Melanie Tory, and Pourang Irani. 2016. The frustrations and benefits of mobile device usage in the home when co-present with family members. In Proceedings of the 2016 ACM conference on designing interactive systems. 1315–1327.Google ScholarDigital Library
- Sunjeong Park and Youn-kyung Lim. 2020. Investigating user expectations on the roles of family-shared AI speakers. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1–13.Google ScholarDigital Library
- Rui Passinhas, Rui Neto Marinheiro, and Paulo Nunes. 2020. Integration of mobile devices in home automation with use of machine learning for object recognition. In Proceedings of the 10th Euro-American Conference on Telematics and Information Systems. 1–8.Google ScholarDigital Library
- Fabio Paternò. 2021. When Artificial Intelligence Alone is not Enough: End-User Creation and Control of Daily Automations. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems. 1–3.Google ScholarDigital Library
- James Pierce. 2019. Smart home security cameras and shifting lines of creepiness: A design-led inquiry. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 1–14.Google ScholarDigital Library
- James Pierce, Claire Weizenegger, Parag Nandi, Isha Agarwal, Gwenna Gram, Jade Hurrle, Hannah Liao, Betty Lo, Aaron Park, Aivy Phan, 2022. Addressing Adjacent Actor Privacy: Designing for Bystanders, Co-Users, and Surveilled Subjects of Smart Home Cameras. In Designing Interactive Systems Conference. 26–40.Google ScholarDigital Library
- James Pierce, Richmond Y Wong, and Nick Merrill. 2020. Sensor illumination: Exploring design qualities and ethical implications of smart cameras and image/video analytics. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1–19.Google ScholarDigital Library
- John Rieman. 1993. The diary study: a workplace-oriented research tool to guide laboratory efforts. In Proceedings of the INTERACT’93 and CHI’93 conference on Human factors in computing systems. 321–326.Google ScholarDigital Library
- Lorena Rossi, Alberto Belli, Adelmo De Santis, Claudia Diamantini, Emanuele Frontoni, Ennio Gambi, Lorenzo Palma, Luca Pernini, Paola Pierleoni, Domenico Potena, 2014. Interoperability issues among smart home technological frameworks. In 2014 IEEE/ASME 10th International Conference on Mechatronic and Embedded Systems and Applications (MESA). IEEE, 1–7.Google ScholarCross Ref
- Quentin Roy, Futian Zhang, and Daniel Vogel. 2019. Automation accuracy is good, but high controllability may be better. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 1–8.Google ScholarDigital Library
- Olivia De Ruyck, Peter Conradie, Lieven De Marez, and Jelle Saldien. 2019. User needs in smart homes: changing needs according to life cycles and the impact on designing smart home solutions. In IFIP Conference on Human-Computer Interaction. Springer, 536–551.Google ScholarDigital Library
- Antti Salovaara, Andrea Bellucci, Andrea Vianello, and Giulio Jacucci. 2021. Programmable smart home toolkits should better address households’ social needs. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1–14.Google ScholarDigital Library
- Fabian Schlenke, Fabian Kohlmorgen, Jörg Bauer, Markus Kuller, Nursi Karaoglan, and Hendrik Wöhrle. 2021. Towards activity recognition in smart homes using multimodal data. In 2021 IEEE 4th International Conference and Workshop Óbuda on Electrical and Power Engineering (CANDO-EPE). IEEE, 25–30.Google ScholarCross Ref
- Scratch. 2022. Imagine, program, share. https://scratch.mit.edu/Google Scholar
- Lei Shi, Maryam Ashoori, Yunfeng Zhang, and Shiri Azenkot. 2018. Knock knock, what’s there: converting passive objects into customizable smart controllers. In Proceedings of the 20th International Conference on Human-Computer Interaction with Mobile Devices and Services. 1–13.Google ScholarDigital Library
- Samarth Singhal, Carman Neustaedter, William Odom, Lyn Bartram, and Yasamin Heshmat. 2018. Time-turner: designing for reflection and remembrance of moments in the home. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. 1–14.Google ScholarDigital Library
- Anselm L Strauss. 1987. Qualitative analysis for social scientists. Cambridge university press.Google Scholar
- Eulim Sull and Youn-kyung Lim. 2018. Designing Health-Promoting Technologies with IoT at Home. In Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems. 1–6.Google ScholarDigital Library
- Kaiwen Sun, Yixin Zou, Jenny Radesky, Christopher Brooks, and Florian Schaub. 2021. Child Safety in the Smart Home: Parents’ Perceptions, Needs, and Mitigation Strategies. Proceedings of the ACM on Human-Computer Interaction 5, CSCW2(2021), 1–41.Google Scholar
- Hongyi Tao and Dhaval Vyas. 2021. DIY Homes: Placemaking in Rural Eco-Homes. In IFIP Conference on Human-Computer Interaction. Springer, 343–364.Google Scholar
- MATÚŠ TOMLEIN, SUDERSHAN BOOVARAGHAVAN, YUVRAJ AGARWAL, and ANIND K DEY. 2018. Supporting Maintenance Operations for IoT-based Activity Recognition using Transfer Learning. (2018).Google Scholar
- Hsiao-Ting Tseng, Hsin-Ginn Hwang, Wei-Yen Hsu, Pei-Chin Chou, and I-Chiu Chang. 2017. IoT-based image recognition system for smart home-delivered meal services. Symmetry 9, 7 (2017), 125.Google ScholarCross Ref
- David Verweij. 2019. Exploring Future IoT for Families through End User Development: Applying Do-It-Together Practises to Reveal Family Dynamics in Technology Adoption. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems. 1–4.Google ScholarDigital Library
- Chao Wang and Pengcheng An. 2021. Explainability via Interactivity? Supporting Nonexperts’ Sensemaking of pre-trained CNN by Interacting with Their Daily Surroundings. In Extended Abstracts of the 2021 Annual Symposium on Computer-Human Interaction in Play. 274–279.Google Scholar
- Barron Webster. 2019. Teachable machine tutorial: Bananameter. https://medium.com/@warronbebster/teachable-machine-tutorial-bananameter-4bfffa765866Google Scholar
- Kristin Williams, Rajitha Pulivarthy, Scott E Hudson, and Jessica Hammer. 2019. Understanding family collaboration around lightweight modification of everyday objects in the home. Proceedings of the ACM on Human-Computer Interaction 3, CSCW(2019), 1–24.Google ScholarDigital Library
- Kristin Williams, Rajitha Pulivarthy, Scott E Hudson, and Jessica Hammer. 2020. The Upcycled Home: Removing barriers to lightweight modification of the home’s everyday objects. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1–13.Google ScholarDigital Library
- Jong-bum Woo and Youn-kyung Lim. 2015. User experience in do-it-yourself-style smart homes. In Proceedings of the 2015 ACM international joint conference on pervasive and ubiquitous computing. 779–790.Google Scholar
- Jong-bum Woo and Youn-kyung Lim. 2020. Routinoscope: Collaborative routine reflection for routine-driven do-it-yourself smart homes. International Journal of Design 14, 3 (2020), 19.Google Scholar
- Jason Wu, Chris Harrison, Jeffrey P Bigham, and Gierad Laput. 2020. Automated Class Discovery and One-Shot Interactions for Acoustic Activity Recognition. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1–14.Google ScholarDigital Library
- Ge Yan, Chao Zhang, Jiadi Wang, Zheng Xu, Jianhui Liu, Jintao Nie, Fangtian Ying, and Cheng Yao. 2022. CamFi: An AI-driven and Camera-based System for Assisting Users in Finding Lost Objects in Multi-Person Scenarios. In CHI Conference on Human Factors in Computing Systems Extended Abstracts. 1–7.Google Scholar
- Hui Ye and Hongbo Fu. 2022. ProGesAR: Mobile AR Prototyping for Proxemic and Gestural Interactions with Real-world IoT Enhanced Spaces. In CHI Conference on Human Factors in Computing Systems. 1–14.Google Scholar
- Zhongyi Zhou and Koji Yatani. 2021. Enhancing Model Assessment in Vision-based Interactive Machine Teaching through Real-time Saliency Map Visualization. In The Adjunct Publication of the 34th Annual ACM Symposium on User Interface Software and Technology. 112–114.Google ScholarDigital Library
Index Terms
- Potential and Challenges of DIY Smart Homes with an ML-intensive Camera Sensor
Recommendations
A review of smart homes-Present state and future challenges
In the era of information technology, the elderly and disabled can be monitored with numerous intelligent devices. Sensors can be implanted into their home for continuous mobility assistance and non-obtrusive disease prevention. Modern sensor-embedded ...
Evolution of Smart Homes for the Elderly
WWW '17 Companion: Proceedings of the 26th International Conference on World Wide Web CompanionSmart home technology provides benefits for the elderly in six primary categories: safety, health and nutrition, physical activity, personal hygiene and care, social engagement, and leisure. Safety is about detecting and mitigating, if not removing, ...
User experience in do-it-yourself-style smart homes
UbiComp '15: Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous ComputingDo-it-yourself (DIY)-style smart home products enable users to create their own smart homes by installing sensors and actuators. DIY smart home products are a potential solution to current problems related to home automation products, such as inflexible ...
Comments