On the Role of Materials Experience for Novel Interactions with Digital Representations of Historical Pop-up and Movable Books

3.1.1 Physical Pop-up and Movable Books.

For interaction with physical books, ten pop-up and movable books were used: one hand-made replica of the (19th century) movable book in the VR experience, two books from the 1960s, one from the 1980s, alongside a range of pop-up books currently in print with more or less elaborate pop-up and movable effects (see Fig. 1 and their details in supplementary materials). They were selected to provide a wide range of material qualities (e.g. varying in stiffness, smoothness), and diverse pop-up and movable effects, such as pull tabs, rotational wheels, and ‘v-fold’, ‘boat’, ‘coil’, and ‘tube’-style pop-ups (terminology from [18]). Eight of ten books were clearly children’s books, while the two others (titled ‘Popville’ and ‘Blue 2’) could also be considered targeting a wider age range based on their topic and illustration style.

3.1.2 Virtual Reality Experience.

The VR experience centres around four historical books from collections held by the National Library of The Netherlands (KB): a pull-tab-style movable book, a double-page pop-up, an accordion book, and a carousel book (n.b. the latter two are strictly speaking neither pop-up nor movable books, according to [70, 121] but they remain relevant to studying interactions with paper materiality) (see Fig. 2 and details in supplementary materials). To create digital representations, the originals were photographed (used as 3D model textures), and their shape reconstructed through 3D modelling. The experience is set in a virtual library environment, where a central open area is surrounded by bookcases. Initially the four books are presented floating in front of the surrounding bookcases, visible at the four cardinal directions, at a 1:1 scale. Books can be selected by grabbing ‘through’ them, at which point they disappear then re-appear enlarged, floating or standing, in the central area. Walking in the virtual space can be done by physical movement or using the controller. Users operate interactive elements (such as turning the page or pulling tabs) using the buttons on the handheld controller. A short, textual description of the selected book is also projected above one of the surrounding bookcases. The four books can be interacted with in different ways. The VR experience runs stand-alone on an Oculus Quest HMD, with its default handheld controllers, and was developed in Unity (see also [78]).

Figure 2:

View Figure

3.6.1 Performativity of Pop-up and Movable Books.

In the observation study, a rich variation of performative actions were observed when interacting with the physical books. These performative actions are summarised in Fig. 3, organised by interactive element, e.g., pull tabs and rotation wheels. We also observed varying temporal patterns associated with different interactions, such as multiple fast repetitions, versus slow or partial actions. A characterisation of these temporal patterns is also added per interaction element in Fig. 3. We observed a more limited set of performative actions, with the pop-up and movable books in the VR experience. These are visualised in Fig. 4, also organised by interactive element, including their temporal characterisations.

3.6.2 Material Qualities of Pop-up and Movable Books.

The material experience characterisations were clustered into five distinct material qualities that are exploited by pop-up and movable books: fold-ability, slide-ability, tear-ability, age-ability, and print-ability qualities; the last two (indirectly) derived from Rudy [109, 110, 111]. The material experience we have associated with the fold-ability and slide-ability qualities rely on a paper’s cut-ability and (sometimes) glue-ability, to allow shaping into endless imaginable configurations (at ‘design’ time). However, we deemed these qualities subordinate to the fold- and slide-ability. The characterisations on the four material experience levels, are summarised per quality in a diagram, such as Fig. 5 illustrates for fold-ability (see others in supplementary materials). The diagram distinguishes characterisations which were mentioned in relation to the physical book (coded in blue and with circles), virtual books (coded in red and with stars), or both (coded in green and with squares).

Fold-ability. This relates to the ability of pop-up and movable books to unfold from a flat stack to a three-dimensional scene triggered by a page turning motion, which participant 3 (P3) described as the book becoming a “3D World that moves when you like slightly open and close the pages”. Fold-ability is associated with emotions such as positive surprise, anticipation (for the next page), joy, and amusement. Moving the pop-ups creates dynamic visual effects and evokes associations of nostalgia, explosions, and “unwrap[ping] the presents” (P10). It triggered performative actions such as repetitive opening and closing, and moving the book and body around in order to further visually explore the book, e.g., by “rotating the book around or rotating myself around to see all the corners” (P3). However, repetitive mechanisms also led to boredom, which in turn led to fast turning or skipping of pages, as illustrated by P21 saying “it’s nice to see how it’s built up, but after three pages, it doesn’t surprise you anymore”.

Slide-ability. This relates to the ability of paper parts to slide in front and behind one another by activating movable mechanisms. This quality is associated with performative actions such as pinching and subsequently pulling, rotating, or ‘wiggling’. Participants experienced it as playful and nostalgic – even recalling specific books from their childhood – and interpreted the effects as inviting, mysterious, and attractive, such as P13 wondering “how far it will spin out” and enticing them to “keep going until it meets the end”. It leads to emotions such as positive surprise, sensory pleasure, amusement, but also fear (of overlooking something), and confusion (what to interact with). While their actions create dynamic visual effects, VR participants emphasise the lack of sound and force feedback. Simplicity and repetitiveness of books also evokes emotions of boredom, as P2 points out that pulling the “string of paper was super underwhelming” stating they didn’t feel they need VR to “just pull a piece of paper.”

Figure 5:

View Figure

Tear-ability. This relates to the quality or rather risk of paper to tear, and thereby be damaged, which was mentioned by many adults in relation to interacting with the physical books. For multiple participants this led to careful performative actions such as partially or slowly turning a page, not applying force, and at times refraining from touching parts of a book. Participants were guided by sensory cues such as feeling friction, hearing scratching paper or by their perception of damage risk, such as P27 posing the question “can I now push through this, or am I breaking it?”. It reinforces their supposition that paper is fragile and that pop-up books are expensive and delicate. This led four participants (with experience in public libraries or their own children) to conclude these books are not (very) suitable for children due to concerns about breakage, as P3 states: “I wouldn’t enjoy it as much [with a little child] because then I would just be terrified like: ‘Don’t break this’ ”. In contrast, others displayed less apprehension (adults and many of the children), engaging in more careless interactions such as flexing the page, pulling the pop-ups apart and other fast and rougher actions, though seemingly not with the intention to break them. Notably, this material quality was absent in the VR experience. P3 explains that: “[the book] in VR cannot really break, and if it breaks it, doesn’t have any […] actual repercussions. You can just reset it”, which stands in contrast to handling the physical books.

Age-ability. This relates to the quality of paper to age. Readers mentioned signs of ageing such as folded edges, faded colours, smell (of old books), changes in sound (crispness), and reduced stiffness, thereby (indirectly) referring to (past) performative actions such as page turning, pinching, and other inherent ageing processes. P16 refers to the aged appearance of the books in VR as “so you see […] the imperfections here”, leading them to conclude originals were scanned rather than it being computer designed. The ageing quality leads to interpretations of items being either old or new, authentic, broken, faded, and fragile. Perceived age also enhances the appeal of interaction, as P6 voices: “It’s always a privilege to be able to touch materials because you’re not always allowed to […]. These […] are quite recent, but if you have […] antiquities […] then you’re not allowed, and that would make it more fun to be able to touch a pop-up book”. The dichotomy between the apparent age of the depictions and the perceived ageing of the material is noted in the virtual books, as “you don’t really get the same kind of the smell of an old book or […] pages being kind of ripped at the edges […] but you still see it in […] the content of the book, that it’s historical.” (P3).

Print-ability. This relates to the ability of paper to be printed, with text, colours, images, and even special effects (e.g. use of metallic and iridescent foils). Performative actions such as turning pages, moving the book around, zooming in, reading, showing and explaining to others, all (indirectly) relate to the print-ability of the paper. Participants frequently commented on the colourfulness of books, as well as on dull or “wrong”(P5 and P12) colours in the VR experience. Print-ability resulted in diverse interpretations, including perceptions of imagery as old-fashioned, beautiful, attention-grabbing, and immersive. P11 highlighted that enlargement in the VR allowed them to “explore and see everything and all the small details because it’s up close, and you can get a lot closer to it than for example [a] real book”. Print-ability triggered sensory pleasure, calmness, and amusement. The use of shiny, metallic, or otherwise textured effects in physical books also elicited positive emotional responses, such as happiness. However, it could also lead to confusion due to the experience being overwhelming, occasionally inhibiting slide-ability and associated performativity: ”Sometimes I’m not entirely sure [...] which thing to pull because some of them have multiple things you can go on. […] I pulled the hat. But, [...] I may have not caught on to that if I [had not] looked closely enough.” (P13).

3.6.3 Other Observations on the Experience and Usability of VR and Physical Books.

Other salient observations from the user study relate to differences in interaction between the physical and virtual books (e.g. their intuitiveness), and differences between users. For physical books, no explanation was needed on the overall working of the books or their mechanisms, even for the youngest participants. In contrast, the VR experience required instructions, as well as extensive and frequent support, to guide nearly all participants through the experience. We also found that in almost all the cases the VR was experienced, participants immediately drew comparisons between the material qualities of the virtual books and physical books (or paper in general), regardless whether they had experienced the physical books before or after the virtual ones (meaning also no order effect was observed in our user study). This might be explained by the fact that nearly all participants had prior experience with pop-up and movable books, which then seems to serve as an implicit reference for the VR experience. A clearly observable distinction between adults and children alongside adults was that the latter led to many more social interactions. With children, the books triggered accompanying adults to read books aloud and pose elaborate, back-and-forth question and answer sessions. Their interactions also focused much more on the content of the books than did solo adults. Several participants commented that they believed the content was targeted toward children. We infer that they largely ignored the content because of this and instead focused more on the form, structure, and dynamics of the books and rarely engage in reading. Adults may also be more conscious of the experimental setting (i.e. the reading aligned with commonly occurring child-adult interactions). Besides the material experience (discussed above), comments about the experience related to: the story in the books, the synergy between story and the interaction mechanisms, the suitability and judgement of the physical and virtual environments (e.g. either liking or disliking the virtual library environment), affordances of VR technology (e.g. allowing you to do less or more than in reality), and usability aspects of the VR experience (e.g. confusion/doubt/frustration with controllers and use cues).

5.1.1 Broadening the Application of the Materials Experience Framework.

The prototype XR experiences of a pop-up and moveable book presented here serve as proofs-of-concept combining and adapting the Materials Experience Framework [47] with associated tools (e.g. [16]) to the Cultural Heritage Human-Computer Interaction (CH-HCI) context. The approach extends the framework’s conventional application area of emerging and/or underdeveloped material development to encompass materials experience characterisation of current and historical materials. This both broadens the framework’s scope and underscores its versatility. We contend its application in CH contexts could be further extended to the characterisation and augmentation of numerous artefact collections and potentially even CH sites (e.g. [96]). As materials experience is also relevant in many other augmented experiences, the relevance of this approach might also extend beyond Cultural Heritage applications and potentially find resonance in diverse domains, such as medical and industrial training applications. Further case studies could also serve to improve and otherwise validate the merits of adapting the material experience framework for CH artefacts and beyond.

5.1.2 Towards a Material Experience Driven Design Methodology for the Cultural Heritage Context.

This paper outlines one possible methodology supporting the design of engaging and meaningful CH experiences: taking the experiential material characterisation of cultural heritage artefacts (in this case pop-up and movable books as a starting point), and using this to inspire novel, digitally-mediated interactions. In our case study, we used observation and interview methods as a means to elicit materials experiences from users (specifically through the use of contemporary surrogates). Following this, we clustered materials experience insights around five key material qualities (following [16]). These were then used in a creative design phase to prototype two XR experiences. Reflecting on our proposed methodology, we acknowledge that there might be other suitable methods for experiential material characterisation (e.g. relying on historical sources, or involving other stakeholders). Furthermore, we envision drawing more parallels to the material driven design (MDD) method [64], such as formulating a (material) vision, and distilling a couple of contrasting or complementary meanings (e.g., ’modest’ and ’provocative’, ’active’ and ’playful’), to guide further concept development. In the CH context, however, this step might be a two-way negotiation, where the narrative(s) might either emerge from the experiential characterisation and/or be predefined (i.e. you want to use an artefact to tell a specific narrative). Further parallels might be drawn to the MDD method, where tinkering with interactions allow designers to sculpt an envisioned materials experience. Finally, a complete methodology would also require a more extensive, iterative approach to prototyping and testing interactions to verify the resulting materials experience.

5.1.3 Expanding Material Experiences Understanding through Digital Technology.

Various materials experiences insights on our user study stem from participants interacting with the earlier-developed VR experience. Observations here proved more telling than viewing participants’ interactions with the physical books alone. We believe this is likely due to the VR experience putting (more) emphasis on certain material aspects otherwise taken for granted or not consciously registered by the participants or designer(s) (e.g. the flexibility of the paper or pages lacking signs of wear). As these provided numerous additional insights usable in the design phase, we contend it would be worthwhile to incorporate such explorations (e.g. in the form of evaluating an early prototype) to surface and characterise unforeseen aspects throughout the design process. Thereby, we advocate utilising (early) prototypes for the experiential characterisation of artefacts and their materials, across all levels, but caution for a too narrow focus, of ‘merely’ evaluating the experience’s (sensorial) realism.

Furthermore, extended reality enables us to capitalise on insights which emerged from the experiential characterisation studies and subsequently creatively manipulate the experiential material qualities. This is a demonstration of direct transferal and augmentation of insights relating to material qualities into novel interaction scenarios. The observations and prototypes above further indicate a broader scope for the MDD method, from using digital augmentations to communicate material qualities in a design process (as defined by e.g. [3]), to the augmentation of material qualities in use time. On top of this, XR technologies present opportunities to modulate spatial and temporal scales, culminating in richer, engaging experiences for users (cf. scaling-up VR pop-up books, and accelerating ageing above). Through XR prototypes we have the ability to fine-tune qualities which intentionally enhance some desirable emotions and associations and, when desired, avoid evoking potential negative associations. One example is the decision to enhance fold-ability by adding dynamic visual effects for the purpose of mitigating boredom evoked by monotonous page turning.

While our interventions have focused on using XR technologies, we believe the materials experience framework leaves open the possibility to design materiality-rooted CH experiences relying on other interaction paradigms. We also acknowledge that there remain fruitful possibilities to craft physical or hybrid interactions with heritage artefacts through other (digital) reproduction methods (e.g. digital fabrication [39, 115]), as well as with the original artefacts themselves through future, non-destructive approaches. Specifically for the case of pop-up and movable, we acknowledge that making physical, paper replicas could be a perfectly suitable solution to experience these artefacts (as-is), and have readers ‘consume’ copies through use. Yet, we believe that the use of digital technology can also offer new opportunities, that would not be attainable with the original or physical replicas, such as dynamically showing process of degradation, or offering spatio-temporally linked contextualisation. Moreover, we believe that using material qualities as inspiration for novel interactions can offer new pathways to intuitively linking the physical and the digital domain (as also discussed in [103]).

5.2.1 Novel Interactions Inspired by (Interrelated) Material Qualities.

Reflection upon the experiential characterisation of pop-up and movable books reveals that material qualities do not function in isolation. Paper’s print-ability is fundamental to the dynamic visual effects, which are in turn essential to the materials experience of the fold-ability and slide-ability qualities (one might imagine all the pop-up and sliding elements were just plain white paper). Likewise, the tear-ability and age-ability of paper are closely connected (e.g. torn books might also be interpreted as being more aged). Understanding, articulating, and orchestrating these interrelations become critical for a designer and serve as a compass guiding the creation of engaging and profoundly meaningful interactions for a specific object or constellation of items. Central to this approach is the acknowledgement of material qualities as a pivotal factor within the design equation. Instead of relying solely on intuitive choices, this approach emphasizes that material qualities can exert a direct influence on decision-making processes in the design process.

5.2.2 Material Performativity and (Additional) Pictorial/Textual Opportunities for Inspiring Augmentations.

This study centred primarily on the material qualities of paper pop-up and movable books, by inviting readers to handle and experiment with them. Such interaction is not common to all historical items; artefacts such as paintings or sculptures are not intended to be handled. While we expect that people’s interactions with heritage artefacts consistently involve all four material experience levels, artefacts crafted from different materials or having different use characteristics demand individual considerations. Additionally, artefacts featuring writing or figurative depictions - such as the books in our case study - may be more conducive to digital augmentation, as they allow leveraging qualities from both their content (i.e. artefacts depicted or described) in addition to the artefact itself. Conversely, objects lacking narrative or figurative components – i.e. the majority of artefacts in (museum) heritage collections – may require additional cues or further considerations for making their materiality intelligible and memorable.

5.2.3 Better Understanding Materials Experience’s Role in Engagement and Learning.

Significant technical and scientific limitations persist in mimicking human touch and other sensory perceptions in digital spaces. Comprehending, quantifying, and augmenting particularly the sense of touch, smell, and taste, are ongoing fields of research (e.g. [89, 98, 129]), and still relatively poorly understood compared to the senses of vision and hearing, as well as other sensory phenomena such as sensory integration (e.g. [27]) and synaesthesia (e.g. [94]). Moreover, a firmer grasp of which elements contribute to people’s understanding and experience of physical object materials would better facilitate digital translations, adaptations, and augmentations.

5.2.4 Ethical Considerations in Designing (Digitally-Mediated) CH Experiences.

The case of augmented materiality provides an example of response to a need to compromise or innovate amid primary concerns of CH institutions. Foremost among these is a tension between the necessity to preserve artefacts and the need for visitors to have engaging and embodied experiences with them. Augmentation necessitates making choices in the design process and represents a certain level of editorial or creative control. Decisions on what to augment and how entail a set of choices for adapting them to a new medium. These choices bring with them further questions about authenticity and faithfulness [26, 68] to original objects on top of authorial/creator intentions, if they are known. Moreover, (re)presentation in any form follows from curational choices, and thereby will always effects our perceptual, interpretive, affective, and performative responses. Whether a representation is therefore ’as-realistic-as-possible’ or authentic in a perceptual or performative sense might be too limiting, or even futile to strive for. Rather, we propose (in line with Penrose [100]) to aim for ’experiential authenticity’, and propose that through our methodology - starting from materials experience characterisation and experiential prototyping of novel interactions - designers can explore alternative pathways to creating experiential authenticity. Along this line, the conclusion whether an experience represents the original artefact, or is a completely new artistic expression, only inspired by a historical artefact, is in part a design choice but also a matter of interpretation, which can differ per stakeholder. Nonetheless, it is important to consider the effect of a digitally-mediated experience on our understanding of the artefact itself, i.e. how the experience might alter our understanding, appreciation, or even care for the original (e.g. as discussed by [25]).

Augmentations rooted in materials experience also introduce more interpretive ambiguity, as opposed to more descriptive and/or factual augmentations. Such ambiguities hold the potential to (co-)design thought-provoking experiences which diverge from the authoritative institutional voice (e.g. [132]) central to debates on inclusivity (e.g. [58]) and decolonisation of CH (e.g. [79, 80]). While opening up the possibility of multiple narratives on artefacts [22], interpretive augmentations confront institutions with challenges on dealing with foregrouding (potentially) controversial CH, which include the potential for – in somebody’s view – misinterpretations, or even abuse of CH.

5.2.5 Practical Considerations in Designing (Digitally-Mediated) CH Experiences.

Digitisation and development of CH experiences remains an expensive and selective process requiring numerous practical considerations related to e.g. the use context and operations (summarised in framework by [30]), many of which not addressed in this study. For our prototypes we decided to use XR technology for its abilities to enable diverse forms of performative interaction, and manipulation of perceptual aspects of the digital representations, which - in the case of mixed reality - could be also overlaid onto its physical counterparts, and controlled through direct, physical manipulation. Yet, designing XR experiences still has many challenges and limitations. For instance, current options for linking physical and virtual is still relatively limited, mostly relying on (unsightly) 2D markers, or bulky trackers. Other forms of computer vision-based detection and tracking are largely restricted/disabled by manufacturers. This, thereby, also (currently) limits options to dynamically synchronise virtual content onto originals or (3D printed) replicas. The complexity of XR development tools (such as Unreal Engine) also poses challenges on conducting fast prototyping cycles, putting practical limits on the amount of design variations that can be explored, which would otherwise enable to better understand how (small) design choices effect the resulting (digitally-mediated) material experience. Moreover, our study and prototyping activities do not address limitations XR technology imposes on social interactions, which are an essential part of a CH experience (pg. 8-10 in [57]). Enabling social interactions, as for instance observed on between the children and their guardians in our study, would require other or additional technological implementations. This lack of shared, immersed perspective was also observed in (preliminary) demonstrations of our mixed-reality prototypes, as for instance the ‘magic’ of the virtual overlays and (the effects of) direct manipulation were not well conveyed to onlookers through a connected screen. Nevertheless, we envision that (social) mixed reality presents fresh opportunities for integration in exhibition settings. It has the potential to address existing limitations in VR by enabling intuitive tangible interaction, real-world presence, and having the potential for both physical and emotional social interaction.

We also acknowledge that our prototypes are not fully-fledged CH experiences, mature for implementation in an exhibition context. The prototypes present an exploration into material experience inspiring digitally-mediated experiences, but meanwhile do not address numerous issues relevant to creating a feasible, viable, and desirable CH experience. The prototypes, for instance, were not designed with a specific audience or exhibition context in mind, nor adhere to all guidelines suggested for (interactive) experiences (e.g. offer different levels of engagement [54], or allow to preview from a distance how it works [31]). Also, we might consider that a (conventional) library offers a different context than a museum, catering to other audiences (e.g. (amateur) researchers, or people seeking a quiet study place) with likely other interests and visiting motivations [41].

We therefore conclude that much more work in this area remains to be done, in designing but also critically evaluating [33, 89] the effectiveness and feasibility of new CH experiences.

5.3.1 Using Contemporary Artefacts as a Proxy for Material Experiences with Historical Artefacts.

Our study utilised contemporary physical (and digital) pop-up books as surrogates due to unavailability of original historical artefacts. This necessitates some caution in interpreting observed materials experiences, in particular regarding surrogates which were in part more contemporary and technologically advanced than the original artefacts. This has the potential to invite critique of the artefacts based on modern norms rather than those of their times. On the other hand surrogates, where available, are inexpensive to work with and for the most part convey the materials experience of the originals. CH artefacts lacking accessible surrogates for reasons such as lack of funds, lack of priority, or copyright may demand alternative approaches to characterising their material experiences.

5.3.2 Familiarity with Artefact Category.

Participants in our study had prior experience and familiarity with pop-up and movable books, which informed their actions, associations and emotional responses. Users’ frames of reference and background knowledge will certainly vary and will likely be more limited for large proportions of heritage collections, potentially leading to more diverse and unguided interpretations. This potentially offered us a clearer and more consistent experiential material characterisation than would be the case for more unfamiliar artefact types.

5.3.3 Incorporating other Stakeholder Perspectives throughout the Design Process.

The scope of our study was solely on ‘use’ time interactions with pop-up and movable books, to the neglect of other possible sources for interpretations on their materials experience. Future studies could do well to broaden the focus to include interpretations of materials experience between different stakeholders, in particular curators and conservation specialists but also authors of paper-engineered books, to elaborate further on the role of materiality in e.g. the design of and care for such artefacts. Concurrently, the CH experiences would benefit from further stakeholder involvement throughout the design process, as advocated in co-design and participatory design approaches for the CH context by others (e.g.[22, 83, 92]).

Finally, the approach taken in this study is more open-ended than definitive. The evaluations of the prototypes on their material experience, by users but also other stakeholders such as curators, are planned as future work by the authors. Increased stakeholder involvement and the development of suitable approaches for (iterative) evaluations are an essential next step in the development of a materials experience rooted design for the CH context. Such evaluations could then potentially also shed light on the role of materials experience in learning and engaging (with CH artefacts) and how best to facilitate this in GLAMs and related settings. Future work focusing on material experiences and creative digital translations of them will continue to support designers in the considerations of augmenting CH artefacts’ material experiences.