Sungbaek Kim
Abstract
As interactive media arts evolve, there is a growing demand for technologies that offer multisensory experiences beyond audiovisual elements in large-scale projection mapping exhibitions. However, traditional methods of providing tactile feedback are impractical in expansive settings due to their bulk and complexity. The EMPop system is the proposed solution, utilizing a straightforward design of electromagnets and permanent magnets making projection mapping more interactive and engaging. Our system is designed to control three permanent magnets individually with one electromagnet by adjusting the current of the electromagnet, reliable and scalable. We assessed its ability to convey directions and the strength of feedback, finding that users correctly identified directions and differentiated feedback intensity levels. Participants enjoyed the realistic and engaging experience, suggesting EMPop’s potential for enriching interactive installations in museums and galleries.
Figure 1: EMPop enhances the multisensory experience of projection mapping by conveying tactile feedback to the surface through repelling neodymium permanent magnets by adjusting the current of an electromagnet.
1 INTRODUCTION
Museums and galleries are increasingly moving away from the traditional ‘Look, don’t touch’ approach embracing interactive and multisensory experiences. This evolution is driven by the understanding that engaging not only sight and hearing but also touch, smell, and more, can significantly enhance visitor’s experiences. Studies by Harvey et al. [4] and Loscos et al. [10] have shown the positive impact of incorporating multisensory elements including haptic feedback in museum environments. Vi et al. [19] further demonstrated the potential of mid-air haptic technology, integrated with sound, to make art more emotionally engaging, underscoring the need for non-invasive, multisensory devices in these settings.
Despite the technological advancements in projection mapping that have transformed audience interaction with art, there remains a significant challenge in integrating senses beyond sight and hearing. Large-scale projection mapping exhibitions, while visually immersive, often do not fully incorporate other sensory experiences like touch, thus limiting the scope of audience engagement. Current solutions such as wearable devices, pseudo-haptics, and VR headsets are often impractical for large-scale use due to their bulk, complexity, and less scalability [11, 12]. Such exhibitions that often feature temporary and variable content require a haptic solution that is flexible, scalable, and able to accommodate changes easily. Furthermore, to truly augment the user experience beyond just the visual and auditory elements, the device needs to provide a good haptic sensation.
In light of these considerations, pin-based haptic devices, such as inFORM [2] and the large-scale tactile display Feeling Fireworks [16], can be seen as suitable solutions. However, these approaches tend to be bulky, complex, and consume significant power. Despite their relative suitability, their substantial size and complexity could disrupt the immersive aesthetic of exhibitions. They may be seen as an obtrusive component rather than an integrated part of the art piece, thus underscoring the need for a more seamless, less intrusive solution. We propose ‘EMPop’, inspired by the electromagnet-based haptic device and pin-array shape-changing display, which focuses on tactile experiences. Unlike traditional methods that use a single electromagnet to create a single actuator, EMPop operates more efficiently by adjusting the current of a single electromagnet to control several permanent magnets individually.
2 RELATED WORKS
2.1 Pin-Based Shape-Changing Displays
Pin-based shape-changing displays provide tactile feedback and dynamic shape changes through the vertical or horizontal movement of pins. Pin arrays in haptic systems play a crucial role in conveying information through touch by creating raised surfaces or textures that can be perceived by the user. Examples of pin-actuated haptic systems include inFORM [2], which uses motorized pins to create dynamic shapes, and Project FEELEX [5], which introduces flexible surfaces that combine haptic sensations with computer graphics. Such displays employ physical changes in shape as a means of input or output for human-computer interaction [15] and widely researched by exploring various actuation mechanisms and interaction scenarios [6, 7, 9, 14, 17, 21]. EMPop shares similarities with these works, as it actuates pins to create haptic feedback, although it is not primarily a shape-changing display.
2.2 Electromagnet and permanent magnet
Electromagnets have been heavily explored for their ease of array organization and the ability to control objects using magnetic forces. In particular, there have been various explorations using the repulsion of electromagnets and permanent magnets. Actuated Workbench [13] proposed the possibility of utilizing electromagnets and permanent magnets by creating an interface where an electromagnet array controls the movement of an object equipped with a permanent magnet. ForceForm [18] created a deformable interactive surface using electromagnet arrays and permanent magnets. Keep in Touch [22] used pot magnets to shorten the distance between permanent magnets and demonstrated its potential as a haptic feedback device. However, these works have limitations that can only control a single permanent magnet with a single electromagnet.
2.3 Projection Mapping on Pin-Based Shape-Changing Displays
Combining projection mapping with shape-changing displays allows for dynamic physical affordances and haptic feedback, enhancing the interaction between the user and the system. FEELEX [5] was one of the earliest shape displays that combined haptic sensations with computer graphics on a tabletop surface. The system used a flexible screen, an actuator array, and a projector to create a spatially continuous surface that users could touch and feel. Sublimate [8] integrated motorized actuators and 3D spatial graphics to explore the computational transition between physical shape output and virtual states. In addition, pin-based shape-changing displays, along with projection mapping, offer unique interaction possibilities and a wide range of applications [1, 2, 9]. EMPop extends the concept of pin-based shape-changing displays to projection mapping, using a pin-based design to deliver tactile feedback that corresponds to the projected content.
3 IMPLEMENTATION
3.1 Module Design
Figure 2: Configuration of the module and how it works. By adjusting the gap between the permanent magnets and the electromagnet and the current of the electromagnet, the three permanent magnets can be controlled individually and further expanded into an array.
The module is designed to use one electromagnet to control three permanent magnets individually. The module design allows the placement of an optimized size and number of permanent magnets within a limited size by the size of the electromagnet’s adsorption plane. This module design allows multiple permanent magnets to be controlled individually by adjusting the current in the electromagnet and the gap between the magnets and the electromagnet.
Magnet Arrangement: The spacing between magnets is determined to be 12.75 mm through flux density simulations to establish the appropriate distance between permanent magnets. Flux density (T), indicative of magnetic force strength, helps in understanding the force over a distance. The EMPop employs a commonly available N35 neodymium magnet (Hc=955,000 A/m, Br=1.22 T) with a diameter of 5 mm and length of 15 mm. Magnetic force becomes negligible at a distance of about 13 mm from another magnet. This aligns with findings from Mechamagnet [23], which noted negligible magnetic interaction for a 3 mm N48 magnet at 9.5 mm away. A reversed magnet in the center further minimizes potential magnetic attraction when the magnets bounce.
Repulsive Actuation: The electromagnet’s magnetic force is adjustable through current modulation, allowing for individual control of each permanent magnet. Operating at 12 V and 0.83 A with a suction power of 500 N, the electromagnet utilizes currents of 0.66 A, 0.76 A, and 0.83 A to control the three permanent magnets sequentially. When a permanent magnet is 1.5 mm above its hole, it can jump in intervals of up to 50 ms. Using Coulomb’s law, the repulsive force during magnet pop-up is estimated at 0.5 N, within the human perceptible range [20].
Gap: For individual control of the permanent magnets, not only the current but also the gap adjustment is required. The farther away the permanent magnet is from the electromagnet, the less current will repel them. The minimum gap was determined to be 3 mm. Though a repulsive force exists within this gap, magnets will stick together if placed too closely. Similar challenges were noted by ForceForm [18] and they placed a 2 mm sheet. The magnets begin to repel at a 3 mm gap and can jump up to 17 cm, with the repelling distance decreasing for larger gaps. Therefore, the three permanent magnets are spaced 0 mm, 3 mm, and 6 mm apart from a gap of 3 mm. This allows them to repel at 0.66 A, 0.76 A, and 0.86 A respectively when the current is increased.
3.2 System Design
We utilized the DMX control system to expand a single module of EMPop into an EMPop array. The DMX control system regulates the magnetic field strength of an electromagnet by converting DMX 512 digital signals into PWM(Pulse Width Modulation) signals and controlling the current of the DC power source [13]. The System delivers tactile feedback for various contents by converting input sources from video or webcam into DMX signals through commercial software. A notable feature of this system is its scalability and versatility; a single DMX decoder can control up to 30 electromagnets, and expanding the system is as simple as adding more Artnet controllers and DMX decoders. While the initial demonstration used eight electromagnets and was potentially bulky at the moment, the system is effective when managing more than 30 modules due to its scalable and efficient design.
4 USE CASE SCENARIO
Figure 3: Types of the use case scenarios. (a) Abstract (The Sun); (b) Realistic (Firework); (c) Content-based (Koi); (d) Interactive.
We present four use case scenarios as EMPop’s application. These example contents show that the EMPop can be adapted to various media art contents dynamically. The scenarios consist of passive and active content. Passive content refers to pre-made control sequences, directly created in Max/MSP, which manage the DMX control system and deliver force feedback. The pre-programmed DMX sequence plays in sync with the video, providing audiences with an amplified sensation upon touch. Passive content can be Abstract or Realistic(Figure 3a, 3b), depending on the feedback method used. Active content controls the DMX Control System directly with input from video sources using Touch Designer and Resolume Arena. This content dynamically responds to color changes in the audience input or video source. It is further categorized into Content-Based and Interactive(Figure 3c, 3d) types.
5 USER STUDY
We conducted a user study to assess the effectiveness of the EMPop system’s haptic feedback and the subjective user experience. We focused on assessing participant’s ability to discern and perceive the direction and strength of the haptic feedback and overall experience with EMPop using four prepared use case scenarios. Six participants, aged 25 to 30, were recruited. Their engagement with various content types on the EMPop device was monitored for three minutes.
The EMPop system demonstrated high effectiveness in three key areas: direction detection, strength perception, and user experience. Direction detection was highly accurate (75% to 95.83%), indicating the system’s ability to provide clear directional haptic cues. In strength perception, users consistently recognized three different strength levels, with no variance in perception for the lowest level, showcasing EMPop’s consistent delivery of haptic experiences. User experience evaluation through ‘Quality of Experience (QoE)’ [3] was positive, with 89% of participants rating it as ‘Very Good’ or ‘Excellent’, appreciating the clear haptic feedback and its enhancement of projection mapping experience. The interviews show the novelty of the tactile sensations of EMPop, with many describing it as a ‘whole new’ experience.
6 CONCLUSION
EMPop seeks to overcome certain limitations inherent in current haptic devices within the media art sector, offering a solution that is simple, reliable, and scalable. We aim to enhance user experiences through a novel sensory interface that interacts directly with bare hands, without the need for additional equipment. While both the hardware and software aspects of EMPop hold potential for customization and expansion, it is understood that further development and research are necessary to fully realize its capabilities. This includes improving its resolution and strength and further reducing operational noise when hit. For future development, there are plans to make the system more user-friendly by developing an authoring tool and potential for larger installations, such as tables, walls, or floors, which could simulate various tactile experiences like the sensation of walking on grass. We hope that EMPop will be a step forward in expanding the sensory horizons of media art, even though it remains a subject for ongoing exploration and refinement.
ACKNOWLEDGMENTS
This work was supported by the National Research Foundation of Korea Grant (NRF-2023-RS-2023-00229653). We sincerely appreciate advisor Sang Ho Yoon and colleagues for their contribution.
Footnotes
⁎ Both authors contributed equally to this research.
† Corresponding Author
Supplemental Material
- Sean Follmer, Daniel Leithinger, Alex Olwal, Nadia Cheng, and Hiroshi Ishii. 2012. Jamming user interfaces: programmable particle stiffness and sensing for malleable and shape-changing devices. In Proceedings of the 25th annual ACM symposium on User interface software and technology. ACM, Cambridge Massachusetts USA, 519–528. https://doi.org/10.1145/2380116.2380181Google Scholar
Digital Library
- Sean Follmer, Daniel Leithinger, Alex Olwal, Akimitsu Hogge, and Hiroshi Ishii. 2013. InFORM: Dynamic Physical Affordances and Constraints through Shape and Object Actuation. In Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology (St. Andrews, Scotland, United Kingdom) (UIST ’13). Association for Computing Machinery, New York, NY, USA, 417–426. https://doi.org/10.1145/2501988.2502032Google ScholarDigital Library
- Abdelwahab Hamam, Abdulmotaleb El Saddik, and Jihad Alja’am. 2014. A Quality of Experience Model for Haptic Virtual Environments. ACM Trans. Multimedia Comput. Commun. Appl. 10, 3, Article 28 (apr 2014), 23 pages. https://doi.org/10.1145/2540991Google ScholarDigital Library
- Mark L. Harvey, Ross J. Loomis, Paul A. Bell, and Margaret Marino. 1998. The Influence of Museum Exhibit Design on Immersion and Psychological Flow. Environment and Behavior 30, 5 (1998), 601–627. https://doi.org/10.1177/001391659803000502 arXiv:https://doi.org/10.1177/001391659803000502Google ScholarCross Ref
- Hiroo Iwata, Hiroaki Yano, Fumitaka Nakaizumi, and Ryo Kawamura. 2001. Project FEELEX: adding haptic surface to graphics. In Proceedings of the 28th annual conference on Computer graphics and interactive techniques(SIGGRAPH ’01). Association for Computing Machinery, New York, NY, USA, 469–476. https://doi.org/10.1145/383259.383314Google ScholarDigital Library
- Jingun Jung, Eunhye Youn, and Geehyuk Lee. 2017. PinPad: Touchpad Interaction with Fast and High-Resolution Tactile Output. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems(CHI ’17). Association for Computing Machinery, New York, NY, USA, 2416–2425. https://doi.org/10.1145/3025453.3025971Google ScholarDigital Library
- Byoungjae Kim, Jiwoo Hong, and Woohun Lee. 2022. Poly: Shape-changing Conversational Agent Helps Identify Multiple Characters in Storytelling. In Sixteenth International Conference on Tangible, Embedded, and Embodied Interaction. ACM, Daejeon Republic of Korea, 1–7. https://doi.org/10.1145/3490149.3505573Google ScholarDigital Library
- Daniel Leithinger, Sean Follmer, Alex Olwal, Samuel Luescher, Akimitsu Hogge, Jinha Lee, and Hiroshi Ishii. 2013. Sublimate: state-changing virtual and physical rendering to augment interaction with shape displays. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, Paris France, 1441–1450. https://doi.org/10.1145/2470654.2466191Google ScholarDigital Library
- Daniel Leithinger and Hiroshi Ishii. 2010. Relief: a scalable actuated shape display. In Proceedings of the fourth international conference on Tangible, embedded, and embodied interaction. ACM, Cambridge Massachusetts USA, 221–222. https://doi.org/10.1145/1709886.1709928Google ScholarDigital Library
- Céline Loscos, Franco Tecchia, Antonio Frisoli, Marcello Carrozzino, Hila Ritter Widenfeld, David Swapp, and Massimo Bergamasco. 2004. The Museum of Pure Form: touching real statues in an immersive virtual museum.. In VAST. 271–279.Google Scholar
- Mille Skovhus Lunding, Germán Leiva, Jens Emil Sloth Grønbæk, and Marianne Graves Petersen. 2022. ProjectAR: Rapid Prototyping of Projection Mapping with Mobile Augmented Reality. In Adjunct Proceedings of the 2022 Nordic Human-Computer Interaction Conference (Aarhus, Denmark) (NordiCHI ’22). Association for Computing Machinery, New York, NY, USA, Article 55, 5 pages. https://doi.org/10.1145/3547522.3547679Google ScholarDigital Library
- Y. Miyatake, T. Hiraki, D. Iwai, and K. Sato. 2023. HaptoMapping: Visuo-Haptic Augmented Reality by Embedding User-Imperceptible Tactile Display Control Signals in a Projected Image. IEEE Transactions on Visualization & Computer Graphics 29, 04 (apr 2023), 2005–2019. https://doi.org/10.1109/TVCG.2021.3136214Google ScholarDigital Library
- Gian Pangaro, Dan Maynes-aminzade, and Hiroshi Ishii. 2002. The Actuated Workbench: Computer-Controlled Actuation in Tabletop Tangible Interfaces. ACM Transactions on Graphics (11 2002). https://doi.org/10.1145/1201775.882330Google ScholarDigital Library
- Ivan Poupyrev, Tatsushi Nashida, Shigeaki Maruyama, Jun Rekimoto, and Yasufumi Yamaji. 2004. Lumen: interactive visual and shape display for calm computing. In ACM SIGGRAPH 2004 Emerging technologies on - SIGGRAPH ’04. ACM Press, Los Angeles, California, 17. https://doi.org/10.1145/1186155.1186173Google ScholarDigital Library
- Majken K. Rasmussen, Esben W. Pedersen, Marianne G. Petersen, and Kasper Hornbæk. 2012. Shape-changing interfaces: a review of the design space and open research questions. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, Austin Texas USA, 735–744. https://doi.org/10.1145/2207676.2207781Google ScholarDigital Library
- Dorothea Reusser, Espen Knoop, Roland Siegwart, and Paul Beardsley. 2019. Feeling fireworks: An inclusive tactile firework display. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 1–11.Google ScholarDigital Library
- Alexa F. Siu, Eric J. Gonzalez, Shenli Yuan, Jason B. Ginsberg, and Sean Follmer. 2018. shapeShift: 2D Spatial Manipulation and Self-Actuation of Tabletop Shape Displays for Tangible and Haptic Interaction. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, Montreal QC Canada, 1–13. https://doi.org/10.1145/3173574.3173865Google ScholarDigital Library
- Jessica Tsimeris, Colin Dedman, Michael Broughton, and Tom Gedeon. 2013. ForceForm: a dynamically deformable interactive surface. In Proceedings of the 2013 ACM international conference on Interactive tabletops and surfaces. 175–178.Google ScholarDigital Library
- Chi Thanh Vi, Damien Ablart, Elia Gatti, Carlos Velasco, and Marianna Obrist. 2017. Not just seeing, but also feeling art: Mid-air haptic experiences integrated in a multisensory art exhibition. International Journal of Human-Computer Studies 108 (Dec. 2017), 1–14. https://doi.org/10.1016/j.ijhcs.2017.06.004Google ScholarDigital Library
- Wenzhen Yang, Jinpen Huang, Ruirui Wang, Wen Zhang, Haitao Liu, and Jianliang Xiao. 2021. A survey on tactile displays for visually impaired people. IEEE Transactions on Haptics 14, 4 (2021), 712–721.Google ScholarDigital Library
- Shigeo Yoshida, Yuqian Sun, and Hideaki Kuzuoka. 2020. PoCoPo: Handheld Pin-based Shape Display for Haptic Rendering in Virtual Reality. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. ACM, Honolulu HI USA, 1–13. https://doi.org/10.1145/3313831.3376358Google ScholarDigital Library
- Juan Jose Zarate, Olexandr Gudozhnik, Anthony Sébastien Ruch, and Herbert Shea. 2017. Keep in Touch: Portable Haptic Display with 192 High Speed Taxels. In Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems (Denver, Colorado, USA) (CHI EA ’17). Association for Computing Machinery, New York, NY, USA, 349–352. https://doi.org/10.1145/3027063.3052957Google ScholarDigital Library
- Clement Zheng, Jeeeun Kim, Daniel Leithinger, Mark D. Gross, and Ellen Yi-Luen Do. 2019. Mechamagnets: Designing and Fabricating Haptic and Functional Physical Inputs with Embedded Magnets. In Proceedings of the Thirteenth International Conference on Tangible, Embedded, and Embodied Interaction (Tempe, Arizona, USA) (TEI ’19). Association for Computing Machinery, New York, NY, USA, 325–334. https://doi.org/10.1145/3294109.3295622Google ScholarDigital Library
Index Terms
- EMPop: Pin Based Electromagnetic Actuation for Projection Mapping
Recommendations
Using Handmade Controllers for Interactive Projection Mapping
MM '15: Proceedings of the 23rd ACM international conference on MultimediaIn this short paper I will look at the use of hand-made gaming controllers for my own interactive artworks, in particular the technology behind the interactive projection mapping artwork Shadows blister those who try to touch, exhibited at ACM MM 2015 ...
Towards Understanding the Design of Positive Pre-sleep Through a Neurofeedback Artistic Experience
CHI '19: Proceedings of the 2019 CHI Conference on Human Factors in Computing SystemsPoor sleep has been acknowledged as an increasingly prevalent global health concern, however, how to design for promoting sleep is relatively underexplored. We propose neurofeedback technology may potentially facilitate restfulness and sleep onset, and ...
Digital arts: did you feel that?
CHI EA '13: CHI '13 Extended Abstracts on Human Factors in Computing SystemsThis panel considers the relationships between the interactive arts, audience engagement and experience design. What might each offer the other? Engagement and experience are central to current HCI thinking. We will present and argue about the research ...
Comments