Investigating the crowd-drawing effect, on passersby, of pseudo-crowds using multiple robots

Due to their conversational skills, service robots are expected to replace posters or digital signage as a new advertising medium. These robots must achieve two goals to obtain maximum advertising opportunities in a public space: they must stop as many passersby as possible and make passersby stay as long as possible. In previous studies, to achieve these goals, the robots called out to each passerby one by one, which was inefficient as an advertising medium. Therefore, to obtain more advertising opportunities, this study proposes a method to simultaneously stop many passersby and make them stay, by creating a pseudo-crowd with multiple robots. The pseudo-crowd is a system in which multiple robots acting as an audience are placed around a robot providing information. We verified the effectiveness of the proposed method in a shopping mall. As a result, increasing the number of robots made more passersby stop, and the use of audience robots extended the time passersby spent in front of the robots. Our findings will contribute to the application of robots as a more useful advertising medium in public spaces. GRAPHICAL ABSTRACT


Introduction
Some countries are facing seriously declining birth rates and aging populations. If the current situation continues, it will significantly impact the economy [1]. To cope with this problem, robots are expected to become collaborative partners with humans. Studies on the use of service robots in public places are actively being conducted. In particular, robots that provide advertising information in public spaces (hereafter referred to as 'advertising robots') are in the spotlight as an effective tool for advertising [2]. The advertising robots can be used not only in a commercial context but also in a wide range of applications, such as requesting hand sanitizing. Advertising robots can interact with passersby using their conversational skills, something a conventional advertising medium does not have. In addition, because of the robot's embodiment, advertising robots can attract people's attention and direct their interest to the advertising information [3]. A previous study reported that advertising robots are more effective at attracting users' interest and promoting users' consumption behaviour than conventional advertising media such as tablets [4].
Obtaining many advertising opportunities is essential for advertising robots to perform their roles effectively. In contrast to advertisements by posters, advertisements CONTACT Joichiro Amada is0392ff@ed.ritsumei.ac.jp by robots are not realized without passersby stopping and interacting with the advertising robots. We consider that the robots must complete two steps to obtain many advertising opportunities in a public space: the robots must stop as many passersby as possible and make those passersby stay in front of them for as long as possible. Previous studies have conducted experiments to attract passersby and suggested effective ways for robots to call out to people [4,5]. However, in these studies, the robots interacted with a single passerby or a single group of passersby by calling out to them; thus, this interaction method lost advertising opportunities in public spaces where tens of thousands of people pass by each day because the robot cannot call to other people while interacting with users. In order to efficiently obtain advertising opportunities, it is important to propose an interaction method that prevents the loss of advertising opportunities by attracting other passersby while the robot interacts with a single person (or a single group). Therefore, this study proposes a robot system that can efficiently present information to a large number of passersby in order to prevent the loss of advertising opportunities in a public space such as a shopping mall. A simple method for interacting with more than one passerby (or more than one group) to provide information is to place robots all around a shopping mall. However, such a method cannot attract passersby who are not originally interested in the robot. Accordingly, we assume that this method can only approach a limited number of people, and so loses advertising opportunities. Therefore, what is required is a method in which the robots can simultaneously provide information to a large number of passersby and attract even those who initially have no interest.
Another approach to attracting passersby is to use the effect of the crowd. It is known that when people are interacting with technology such as digital signage in public places, bystanders also try to interact with it. This effect is generally called the Honeypot Effect [6][7][8]. It is also known that when passersby see a crowd of people, they tend to stop and take the same action as the crowd [9]. We consider that forming a crowd around an advertising robot can make passersby stop without directly calling out to them. In other words, the robot combined with the crowd efficiently obtains advertising opportunities. We consider that we can promote the creation of advertising opportunities by the following repetitive process: (a) first, a crowd is formed around the advertising robot, (b) the crowd makes passersby stop, (c) the stopped passersby maintain the crowd, (b') the maintained crowd makes other passersby stop, (c') the new stopped passersby maintain the crowd more, and so on. The important point for achieving this repeated process is (a): forming the first crowd. Here, humanoid-type robots, robots with humanlike appearances, are expected to achieve the same crowd effect in human-robot interaction as would be achieved in human-human interaction [10]. We consider that placing robots acting as an audience around an advertising robot can form a pseudo-crowd as the first crowd. This pseudo-crowd may be useful to make passersby stop and listen to the advertising robot for a long time. In addition, multiple robots representing the pseudo-crowd may generate the Honeypot Effect, that is to say, the crowd situation will attract passersby who are not interested in a single robot. This has the potential to create more advertising opportunities than simply placing multiple robots in a shopping mall. However, it is unclear whether this effect can actually be achieved with a pseudo-crowd formed of robots.
Therefore, this study aims to propose a method to make passersby stop and stay longer more efficiently, by creating a pseudo-crowd with multiple robots. This method is expected to provide more efficient advertising. Furthermore, it is expected that people who are attracted to the pseudo-crowd will create a real crowd and attract other people more efficiently. In order to confirm the effectiveness of the proposed method, we developed an advertising robot system representing the pseudo-crowd with multiple robots. We installed the robots in a shopping mall and measured the behaviour of passersby changed depending on the robot's calls. We verified the effect of the pseudo-crowd with multiple robots in attracting passersby. In particular, we consider that pseudo-crows can be divided into two components: the number of robots and the presence of robots that behave as the audience. We examined the effects of these factors on the behaviour of passersby.
This paper is organized as follows: Section 2 describes related works in the research area of human-robot interaction (HRI), Section 3 describes a method for forming a pseudo-crowd using multiple robots, Section 4 provides an overview of the experiment in a shopping mall, and Section 5 describes the analysis results of the experiment, which are then discussed in Section 6. Section 7 explains the limitations and conclusions of this study.

Related works
In the research area of human-robot interaction (HRI), there are many examples of experiments related to placing advertising robots in public spaces [2,4,5,[11][12][13].
Kanda et al. developed a communication robot for use in a shopping mall to provide shopping information and offer route guidance [11]. The results of an experiment showed that many passersby interacted with the communication robots, which promoted passersby's shopping activities. Shi et al. created a robot for distributing flyers to passersby [12]. They developed the robot's interaction strategy for flyer distribution in reference to the observed behaviour of humans distributing flyers. Brenman et al. investigated the performance of a humanoid service robot versus a tablet service kiosk in the task of guiding passersby to stores and selling products [4]. They placed robots and tablets in stores, used them to call out to passersby, and verified how many passersby the robot and the tablet could stop and guide into the store. Okafuji et al. developed a robot providing shopping information and performing voucher distribution [5]. The robot stopped passersby with various calls, and they verified how well the robot could provide information to passersby, compared to human advertizers. Sakamoto et al. proposed the concept of passive-social interaction of robots, in which they behave as if they are talking together, without talking to passersby directly [13]. They conducted an experiment in a train station, and the results showed that using robots as a passivesocial medium is effective for advertising.
Most robot systems in these studies are designed to interact with a single passerby or a group of passersby. These interaction systems lose advertising opportunities in public spaces where tens of thousands of people pass by each day, because the robot cannot call to new people while interacting with users. To the best of our knowledge, there have been no experiments relating to interaction with more than one passerby (or more than one group) for providing information in public spaces. This is assumed to be because technical limitations make it difficult to interact with multiple passersby. For example, it is more difficult for a robot to recognize the speech of multiple people than a single person. On the other hand, methods that do not directly interact with the users, such as passive-social interaction, have the potential to make multiple people listen to the robot's talk at the same time [13]. While a previous study has achieved passive-social interaction through two robots talking to each other, we form a pseudo-crowd by placing robots behaving as an audience around an advertising robot. The aim of this is that the robot system can attract passersby by the Honeypot Effect without directly calling out to them.

Robot system forming a pseudo-crowd
This section describes a proposed method for forming a pseudo-crowd using multiple robots. We assume that there are two requirements for the pseudo-crowd: a sufficient number of robots and robots behaving as an audience. In the formation of a pseudo-crowd, multiple robots need to be recognized as a crowd by passersby. According to a previous study, three or more robots are necessary to be recognized as a group [14]. Therefore, a pseudo-crowd requires at least this many robots. We use seven robots to form a pseudo-crowd in this study. In addition, we assume that the pseudo-crowd must behave like an audience, because a crowd is a group of people gathered to see something. Therefore, we consider that robots behaving as an audience (hereinafter referred to as 'audience robots') must be installed for multiple robots to be recognized as a crowd. In this study, the proposed robot system forming the pseudo-crowd includes a sufficient number of robots and has audience robots, thus satisfying the two requirements that we suppose must be met to realize efficient advertising by robots, as described in Section 1.
In this study, we assume a situation in which a robot behaving as a speaker (hereinafter referred to as a 'speaker robot') provides information to accompany an advertising video shown on a display, at a shopping mall. The aim of this robot system is to make passersby stop in front of the robot, and stay longer, so that they understand the advertisement video on the display. Proposed robot system. The orange robot is a speaker robot to provide information using a display, and the blue robots act as an audience, which creates a pseudo-crowd. This image shows condition SA-8, as described in Section 4.

System configuration
The robot system used in this study consists of multiple robots and a display, placed on top of a desk, and a sensor, as shown in Figure 1. The speaker robot provides an advertisement, accompanied by the auxiliary display placed on the center of the desk, the audience robots react to the speaker robot, and the sensor detects the position of passersby. The type of robot used is Sota, which is a humanoid robot manufactured by Vstone Co., Ltd. There are eight robots in this system: one orange, which is the speaker robot, and seven blue, which are the audience robots.

Speaker robot
The speaker robot is positioned with its back to the display. The robot provides advertising information to accompany the video, regardless of whether or not a passerby is standing in front of the robot system. When a passerby stops in front of the robots, the audience robots instruct the passerby to listen to the advertisement information, and then the speaker robot draws the attention of passersby to the video, by announcing 'I'm presenting! Please pay attention to this video!'

Audience robots
The audience robots are positioned facing the display. The robots randomly respond to the speaker robot and video, when there are no people present. When a passerby stops in front of the robots, the audience robot closest to the passerby says, 'You can stand here! Let's watch the video together.' Once a robot has talked to a passerby, the robot will not respond until the passerby leaves the area, in order to prevent repeated talking to passersby in a short interval. While one audience robot is talking to passersby, the movements and voices of the other robots are stopped so that the voice can be heard clearly. When a passerby stays for 15 seconds, the audience robot close to the passerby says 'Let's keep watching' to the passerby. In this case, the voice and movements of other audience robots are stopped, but the speaker robot continues to provide advertising information to keep the passersby entertained. Until the robot talks to the passersby again, the audience robots continue responding to the speaker robot and the video.

Sensor
An RGB-D camera (ZED 2 Stereo Camera) is placed behind the table to observe passersby. The field of view (horizontal × vertical) is 110 • × 70 • and the maximum range of the depth sensor is 20 m. We used OpenPose [15], which is a human skeletal detection library, to detect the position of a passerby's face, as seen from the robot. Passerby's face position in the image is estimated with a 2D colour image. Next, the 3D position of the passerby's face is obtained from the coordinates of the point cloud corresponding to the obtained face position in the 2D image. From the 3D position, a passerby is determined to be in front of the robot when the passerby is within 1 m of the robot. If the sensor detects that the passerby is in front of the robots, this automatically triggers a predetermined action: the audience robots instruct passersby to listen to the advertisement information and the speaker robot provides information. This process is executed at 5-15 frames per second. Table 1 shows an example scenario of actual system usage. The robotS' behaviours when there are no people (phase 1) are repeated randomly and continuously until a passerby approaches. The robots' behaviours when a person is staying (phase 2) are repeated randomly every 15 seconds.

Experiment
The aim of this experiment is to verify how the number of robots and the presence of audience robots affect the percentage of passersby who stop and the length of time for which they stay. The robot system described in Section 3 (including the display on the center of the desk) was placed in an aisle in a shopping mall to have passersby watch a promotional video for Ritsumeikan University. This experiment was conducted over five days in October 2021 at a space in AEON MALL Kusatsu, 1 Japan.
We announced to all the passersby through a notification board that this was an experiment, and a video was being recorded along with the sensor data. This study was conducted on an opt-out basis for unwilling participants who wanted to be removed from the video and sensor data. They can delete all data by requesting a nearby experimenter. The opt-out process may have changed the behaviour of passersby, such as the passersby who attempted to interact with the robot, but quit the interaction owing to the notification. However, no one asked to delete the record in the experiment; thus, the effect of opt-out on the experimental results is expected to be minimal.
This study was approved by the Research Ethics Committee of Ritsumeikan University (reference number: BKC-HitoI-2020-027-2).

Conditions
As described in Section 3, we assume that a sufficient number of robots and the presence of audience robots, which are the requirements of the pseudo-crowd, can increase the rate at which passersby stop and the length of time for which they stay, which results in improved advertising opportunities. Therefore, in order to verify the two factors of the pseudo-crowd with multiple robots, we conducted our experiment under four (2 × 2) conditions. Two conditions had the purpose of verifying the effect of the presence of audience robots: 'all robots act as speaker robots' (S condition; Speakers), and 'only one robot acts as a speaker robot and the others act as audience robots' (SA condition; Speaker and Audience). In the S condition, only the orange robot performed the action to draw a passerby's attention to the video when the passerby stopped (described in Section 3.2). The remaining two conditions were to verify the effect of the number of robots. In one condition only two robots were used, and in the other, eight robots were used. In both conditions, one of the robots was orange in appearance, and the others were blue. The conditions are written as (S/SA)-(number of robots); e.g. the condition in which all robots are speakers and the number of robots is two is written as S-2. The pseudo-crowd proposed in this paper corresponds to the SA-8 condition, as shown in Figure 1. Figure 2 shows the robot placement for each of the other conditions. In addition to these four conditions, we also prepared a baseline condition of only one speaker robot, to compare our system to conventional advertising robots. Each condition was implemented for six hours (10am-4pm) a day on holidays in October 2021.
Depending on the order in which a passerby experiences the conditions, the occurrence of the novelty effect [16] or the simple contact effect [17] is conceivable. However, the same passerby is rarely exposed to different conditions because the environment in this experiment is a large suburban shopping mall. It is therefore assumed that there is little influence from order effects.

Measurement
The number of passersby who passed in front of the robot was counted from the recorded video. A passerby who stopped in front of the robot was labeled as 'Stopped,' and the duration which the passerby spent in front of the robot was recorded as 'Staying time.' To ensure valid annotation results, the labeling was performed by three coders. One was the author, J. A., and the others were people unrelated to this study, hired as part-time workers. In order to ensure uniformity in the criteria used for judging the behaviours, e.g. 'Stopped,' the three coders Before approaching the robots, did you think you would enjoy listening to them? Q3 Before approaching the robots, did you feel any awkwardness, i.e. that it was difficult to approach the robots? Q4 While the robots were talking, did you enjoy listening to them? Q5 While the robots were talking, did you feel embarrassed? Q6 While the robots were talking, did you feel any pressure? Q7a Did you watch the video that was playing? Q7b Did you understand the video? determined the criteria together before labeling. 10% of the total data was labeled by all three coders to calculate the matching rate of the labelling. An analysis of the overlapped data by time sampling methods showed that they were well matched (interval 1 s, matching rate .857).
The evaluation was based on two factors: 'Stopped rate (SR),' and 'Staying time (ST).' The SR is the ratio of 'Stopped' to the total number of passersby, which indicates the percentage of passersby that the robot was able to stop. The ST is the duration of 'Staying time' for each passerby labeled 'Stopped. ' Passersby who stopped during the experiment were asked to answer a questionnaire after they terminated the interaction with the robot. In each condition, 30 people were randomly selected and answered the questionnaire shown in Table 2. In the questionnaire, Q1-3 were related to SR, Q4-6 to ST, and Q7 to whether passersby remembered the advertisement. A 5-point Likert scale (1)(2)(3)(4)(5) was used to answer the questionnaire, with 5 being 'yes, very much' and 1 being 'no, not at all.' Q7a was a binary 'yes' or 'no,' and Q7b was answered only when Q7a was 'yes.'

Effect of the number of robots
A previous study has reported that, when compared to the case of using a single robot, using multiple robots attracted the attention of passersby more and increased the stopping rate [13]. We predicted that a larger number of robots would attract more attention and make passersby more likely to stop. Therefore, the following hypothesis was formed regarding the effect of the number of robots. H1: Increasing the number of robots is effective in stopping passersby.
If the SR is higher in the S-8 and SA-8 conditions than in the S-2 and SA-2 conditions, H1 is supported.

Effect of the presence of audience robots
Previous research has shown that passersby are more interested in and more likely to understand what a robot says when two robots talk to each other than when a robot directly talks to the passersby [13,18]. We assume that high interest in what the robots talk about would increase staying time. In this experiment, multiple robots talked with each other, acting as the audience, only in the SA conditions. Therefore, the following hypothesis was formed regarding the effect of the presence of the audience robots.
H2: The presence of an audience robot is effective in increasing staying time.
If the ST is longer in the SA-2 and SA-8 conditions than in the S-8 and S-2 conditions, then H2 is supported. Table 3 shows the results of passerby labeling for each condition. SA-8 had the highest SR. The average SR results according to the number of robots and the presence or absence of audience robots are presented in Figures 3-4. These results were compared using the chisquare test, using Cramer's V as the effect size, and Bonferroni corrections were made for comparisons. The test result of individual pairs of the SR is presented in Table 4, and Bonferroni corrections were made for comparisons. Figure 3 shows that the average SR in the 2-robots condition was nearly doubled in the 8-robots condition, increasing from 10.45% to 18.02%, confirming that many passersby stopped in the 8-robots condition, as shown in Figure 5. The chi-square test revealed significant differences in the average SR between 2-robots and 8-robots conditions (χ 2 (df = 1) = 228.8, p < .001, V = .103). The tests on individual pairs between the numbers of robots showed respectively significant differences, as shown in Table 4. Figure 4 shows the average SR according to the presence or absence of audience robot(s), and the chi-square test revealed no significant differences in   the average SR between these conditions (χ 2 (df = 1) = 0.267, p = .606, V = .004). The tests on individual pairs between the presence/absence of audience robots showed respectively no significant differences, as shown in Table 4. As a reference value, the baseline condition had lower SR than all other conditions.

Result
165 * denotes significant differences at the 0.5% ( = 5%/10) level. Figure 6 shows the average ST according to condition. The average ST was shortest in the S-8 condition and longest in the SA-8 condition. We conducted a Two-factor factorial ANOVA on the conditions. The result showed a significant difference in the main effect relating to the presence or absence of audience robots (F(1, 3228) = 5.18, p = .023). The results show no significant differences in the main effect of the number of robots, and the interaction effects between the number of robots and presence or absence of audience robots (F(1, 3228) = 0.06, p = .808; F(1, 3228) = 3.04, p = .082). As a reference value, the baseline condition had the second-highest average ST.
Next, the results of the questionnaire are presented. Figure 7 shows the average results according to the number of robots, and Figure 8 shows the average results according to the presence or absence of audience robots. The chi-square test was used to compare answers to Q7a, and Mann-Whitney's U test was used for the other questions. Cramer's V was used as the effect size for the chi-square test. The results of the tests are summarized in Table 5. Regarding increasing the number of robots, the results showed that this led to a significant increase in the conspicuousness of the robots (Q1: U = 1324, p = .012), but also a significant decrease in understanding of the video content (Q7b: U = 335.5, p = .047).

Discussion
The average SR results according to the number of robots show that the average SR increased significantly as the number of robots increased; thus H1 was supported. The questionnaire results showed that an increase in the number of robots made the robots stand out more before passersby approached them (Q1). A previous study reported that multiple robots attracted the attention of passersby and stopped them more effectively than a single robot [13]. In this experiment as well, increasing the number of robots made them more conspicuous, and the average SR increased as passersby noticed them. While the previous study compared one and two robots, this study compared two and eight robots, thus, the results imply the generality that increasing the number of robots attracts the attention of passersby regardless of the number of robots. Therefore, increasing the number of robots is effective in making passersby stop.
However, the questionnaire results showed that video understanding decreased significantly as the number of robots increased (Q7b). This is considered to be because too many robots obscured the display, or too much robot chatter made it difficult for passersby to concentrate on the video and listen to the description by the speaker robot. In order to develop an effective advertising robot, we need to design a robot system that does not interfere with the understanding of the video; e.g. by creating an empty space in front of the display when passersby approach the robot. Another reason for   the low degree of understanding was that the passersby paid attention only to the robot and not to the video. If the information was provided through conversation with the robot instead of through a display, the advertisements might be fully understood. Therefore, we need to design a robot system that can appropriately control the interest in the robot itself and the interest in the video.
The ST increased significantly with the presence of audience robots, thus supporting H2. This is considered to be because the passersby were interested in what the audience robots were discussing. Previous research has shown that two robots talking to each other, rather than direct interaction between robots and passersby, can promote users' interest in what the robots talk about [13]. In this experiment, the audience robots reacted to the speaker robot to promote users' interest in what the robots talked about. Another assumed reason for the longer ST is that passersby were interested in the audience robots themselves. Passersby need to pay attention to more objects in the SA condition than in the S condition, in which the only object of attention is the speaker robot. Thus, we consider that passersby required more time to understand the situation in the SA condition, which led to the extended ST. Therefore, the presence of an audience robot is effective in extending stay time.

Limitations and future work
The proposed method of a pseudo-crowd formed of multiple robots was successful in making a real human crowd form around the robots, as shown in Figure 5. This is because of the improvement of the SR due to the increase in the number of robots and the improvement of the ST by the presence of audience robots. We could generate a phenomenon that was not seen in the baseline condition, demonstrating that we could efficiently create advertising opportunities without calling out to each individual. However, it is unclear whether the pseudo-crowd itself actually functions as a crowd, in terms of the Honeypot Effect. In other words, it is unknown whether the passersby stopped because they consciously or unconsciously recognized multiple audience robots as fellow passersby, or simply because the robot system strongly attracted attention. It will be a future task to clarify what caused the behaviour of the passersby observed in this experiment.
To make the system effective as an advertising medium, the interaction methods need to be improved. In this experiment, the robots stopped passersby and provided information with videos as an advertising method. Many people interacted with the robots, but the staying time was short and understanding of the advertisement content was low; thus, we need to change the advertising method after passersby stop in front of the robots. In particular, according to the results of the questionnaire, only approximately half of the stopping passersby watched the video, which means that the video did not fully work as an advertisement in this study. Hence, the interaction method of the advertisement medium should be improved, and a future task will be to find the optimal method, e.g.leaving the space in front of the display empty after passersby approach the robots.
In future work, we need to investigate the influence of the appearance and size of the robots, two further factors that are expected to alter the effect of the pseudo-crowd. It is unclear whether this study would have obtained similar results if a different type of robot was used. In addition, we need to verify how the placement and interaction of the audience robots affect the behaviour of passersby.

Conclusion
The purpose of this study was to verify how elements of a pseudo-crowd using multiple robots affect the behaviour of passersby. We developed an advertising system that uses multiple robots and installed the robots in a shopping mall. In particular, we focused on the number of robots and the presence of audience robots in the pseudo-crowd system and verified how each of these factors affects how successfully passersby are attracted. The results show that an increase in the number of robots attracts more passersby to stop and that the presence of the audience robots increases the staying time of passersby. These results indicate that, in the design of robot systems for advertising in public spaces, the pseudo-crowd with multiple robots increases advertising opportunities. We believe that our findings will contribute to the application of robots as a more useful advertising medium in public spaces.
Jun Baba received his ME degree in informatics from Kyoto University, Kyoto, Japan in 2014. He was a data scientist at CyberAgent, Inc. in Tokyo, Japan from 2014 to 2017. He has been a research scientist at CyberAgent AI Lab and a visiting researcher at Osaka Universit since 2017. His research interests include teleoperation for social robots, human-computer interaction in service encounter, and artificial intelligence.