TOOLS USED IN AMBIENT USER INTERFACES

Research of ambient user interfaces is popular among many researchers. The main idea is that a user interface would react onto its user and would execute reactions on his/her actions. Main problem with creating ambient user interface is that for different domains the human-computer interaction changes. In this article we investigate different tools that are employed in ambient user interface and the different approaches how they are used. We focus mainly on the use of the multi-display set-up, touch screens, kinect, eye-tracking devices and brain-computer interface technology. We propose approaches how they can be utilized in an ambient user interface, how they affect the user and how they can be used for the evaluation of ambient user interface. It is essential for future research that these measures evaluate the efficiency of the use of ambient user interface in different domains. We briefly analyze natural user interface and its association to ambient user interface. In the conclusion we establish the steps for the future research of this topic.


INTRODUCTION
Main goal of computer systems is to lighten the execution of common tasks for its user.The traditional computer work environment of user contains a computer station, output devices, such as a monitor or occasionally a printer, and input devices, such as a keyboard and a computer mouse.Multiple displays can be connected to one computer station, thus increasing the work space of the computer user.Research by Owens et al. [1] suggests that having multiple displays in the work environment enhances the users' productivity.Human-computer interaction in the multi-display environment changes [2].It is required for the user interface of the multi-display work environment to be adapted to match the user expectation of the system behavior [3].Ambient user interfaces are interfaces, in which the information is displayed in the users' periphery, only moving to the center of attention of the system user when desired.Attention of the user is detected through the use of special devices that observe him/her.These devices can identify the point of regard (e.g. point on screen toward which the user is looking), the relative position of the user and the computer in the work environment, the brain waves in users' head or various physical conditions of the system user, such as heartbeat or amount of sweating.The human-computer interaction in ambient user interfaces is distributed among multiple devices [4].
Ambient user interfaces provide the user with comfortable control, that is in the ideal case unobtrusive for the user.Such interfaces are nowadays deployed in ambient intelligence environments [5].In these environments system automatically reacts on the user actions without direct involvement of him/her.This application eases the execution of his/her task without interfering with his/her standard behavior.Although ambient user interfaces have been around for some time already, examining them in different domains is still a challenge for researchers, as there are different approaches to creating and sustaining them [6].
According to the survey made by Sadri [7], ambient in-telligence is used in health care, intelligent households, assisted living, shopping, museums, tourism, education.To test the ambient user interfaces, the researcher needs to either inspect them in their environment or construct a laboratory that is designed according to the needs of the targeted domain.Creating a laboratory for testing the ambient interfaces is challenging, because it needs to be adaptable to the needs of any domain.Evaluation of the ambient user interfaces is especially difficult because of the Hawthorne Effect [8] that states, that behavior of the experiment participant changes as a consequence to the participants awareness of being studied.However, with prolonged exposure to ambient user interface, the Hawthorne Effect seems to diminish.

ANALYSIS
There are three main focuses in the research of natural and ambient computer system that this paper takes into consideration.The first one is the multi-display work environment, which is simple to set-up for different domains and is able to positively affect the productivity of the system user.
The second one is the ambient user interface, which reacts to an user action in his/her environment.
The third one is the natural user interface, which provides the user with human-computer interaction that is unobtrusive and intuitive for him/her.Important aspect of such interface is that it would behave according to the users' understanding of the world, so any user should be able to use it when (s)he comes into contact with it.

Work environment with multiple monitors
Multiple displays are being introduced into different workplaces with the aim of increasing the productivity of the system user [1].There are two environments that use multiple displays.
The first one is multi-monitor work environment, in which multiple monitors are placed next to each other and serve as one large monitor.Benefits of this work environment can also be achieved through the use of a projector.In this case the bezels between each screen are eliminated so better sensation of presented image is achieved.Ebert et al. [9] appoint that it is harder to calibrate such system in order to create a seamless system.Advantage of this work environment is the increased user display space, which can be used for better organization of the work elements.Ebert et al. identify three ways of creating a seamless system.The first one is to use the offset approach, which ignores edges and their effect on continuity of the scene.In this approach the entire screen is displayed, however the screen seems distorted.The second one is the overlay approach, in which the data is processed and displayed onto screens as one continuous image.However the data that would have been shown on the bezels of the screens is not shown.This can lead to loss of data, such as a text that would be shown in this area.The third one is to project missing image information directly onto the edges of monitors.They accomplished the latter one with placing white cardboard on the edges of monitors and displaying missing image on them with projectors.These approaches are shown in Fig. 1.
Fig. 1 Displaying data on multi-monitor with: (a) offset approach, (b) overlay approach, (c) projecting missing image onto the edges of monitors [9] Since the loss of data is critical for the adjustable user interface and the cost of creating a system with projecting image on edges of monitors is high, in a set-up of multimonitor for inspecting different tools in an ambient user interface we recommend the offset approach.
Additionally multi-monitor set-up behaves as single monitor, so moving mouse cursor from one monitor to another does not happen through jumps and is more intuitive for use.
Another approach how the benefits of the multi-monitor work environment can be achieved is through the use of a wide screen environment, in which one screen with the diagonal of at least 29 inches is used.
The second environment that uses multiple displays is multi-display work environment.Displays in this environment do not behave as a single wide screen, so they can be put on any place.Advantage of the multi-display work environment is its simple set-up.Disadvantage of this environment is that switching focus from one display to another is performed with cursor jumps, as these displays can have different sizes and resolutions [3].Additionally these displays can be placed on different walls of the room.
Drawback of the larger display space (whether through multi-monitor or multi-display environment) is that the user can lose his attention more easily.Wrong distribution of the work elements can lead to decreased productivity.Larger display space allows its user to display non-work related application, which may act like a distraction from work and even further decrease the users' productivity [2].
In our previous work we inspected the ways of observing a software developer in a multi-display environment [10].We detected that some observable patterns of behavior, for example rotating the developers head towards the observed display, can be used to perform human-computer interaction.
Kern et al. [11] focused on problem with attention switching in the multi-display work environment.They address this problem with creating a visual reminder -Gazemark that serves as a visual placeholder of the last area on the screen, on which the user fixated his look.For evaluation of this approach subjective questionnaires were used.For better understanding and measurement of such approaches arises the need to design objective measures, that can be used for evaluation of multi-display work environments and solutions, that address attention switching problems.
Ebert et al. [9] focused on the large display space achieved through the multi-monitor desktop and stereoscopic displays.They created two focus and context screen metaphors that address the problems with multi-monitor systems identified by Ni [12].The first problem is that data displayed on multi-monitor is influenced by the edges of the monitors.The second problem is the loss of data in stereoscopic screens used to display 3D data for viewers.
Set of measures for the evaluation of a multi screen television was established by Vatavu and Mancas [13].They created multi-display set-up with multiple projectors that displayed different screens onto wall.This set-up allowed them to test variety of screen resolutions and screen compositions without the use of different sized monitors.For measuring different layouts of screens they used an eyetracker that tracked the eye-gaze point of the participants.Final set of the objective measures can be used for multidisplay work environment and consist of: • discovery time -the time required for the viewer to make a pass over all screens • discovery sequence -the sequence of screens that was traversed by the viewer's eye gaze during the discovery time • screen watching time -percentage of the visual attention to each screen • transition speed -the average eye gaze speed at which the viewers perform transitions between screens • eye gaze travel distance -the total distance traveled by the viewer's eye gaze • eye gaze travel speed -the average speed of the viewer's eye gaze.
• switch time -percentage of time during which the viewer's eye gaze travels between screens.
Projectors are good solution for testing various sizes of display screens, however the distance between screens and the viewer is constant, which influences the way the viewer eyes fixate on them.However for the ambient user interface we recommend using physical displays as users can cover projectors when moving in the ambient user interface.
Probst et al. [14] designed a distributed display environment as a response to the prolonged sitting that has increased in the recent years and is connected with increased health-related risks.In this environment the user has two monitors connected to a single workstation and can switch from sedentary to stationary work.This solution is both good for the productivity of the system user and his health.This solution can be used in an ambient user interface, as the user can sit and watch the displays, but when he would like to interact with the interface, he would need to stand up so the computer system can observe him.
Another approach to the multi display work environment considers tangible user interfaces.It was examined by Lee et al. [15].In this approach low cost infrared (IR) tracking was used that allowed the users to directly interact with digital products with touch.IR tangibles allow natural user interactions, as users could control the displayed object with touch.This made the objects on the screen behave similarly to a touch screen device.However this solution can be achieved with the use of touch screens as they are becoming more affordable and thus we recommend using a touch screen.Although, this solution can still be useful for high resolution screens and the screens are not getting dirty from the constant touch interaction.

Ambient user interfaces
Ambient user interface surrounds the user and (s)he can interact with it with the use of input-output devices or with his/her actions.Ambient user interface is aware of the user and adapts its behavior according to his/her needs.This adaptation is caused by an artificial intelligence employed in the ambient user interface.Because of this, ambient user interfaces are often called ambient intelligent.In this computing paradigm, the conventional input and output media are no longer present [7].Sensors and processors are integrated into everyday objects in order to support the inhabitants in the interface.Sadri [7] summarized the basic characteristics of an ambient intelligence system: • Context aware -it exploits the contextual and situational information.
• Personalized -it is personalized to the needs of each individual.
• Anticipatory -it anticipates the needs of user without needing mediation • Adaptive -it adapts to the changing needs of individuals • Ubiquity -it is embedded into environment • Transparency -it fades into background of everyday life.
Ambient intelligence is already present in some environments and is still under the research focus.Work of Lee et al. [16] addresses the third dimension of such interface with a head mounted device that calculates the point of regard of the user in space, not only identifying the x and y coordinate of the point of regard, but also the depth depicted by the z coordinate.Kocejko et al. [17] also focused on the head mounted device.They use this device for improving the gaze estimation with the use of the coordinate of the users' head in space.However head mounted solution is obtrusive for the user and is not satisfying the characteristics of the ambient intelligence system.That is why we recommend using another device, such as kinect, for the calculation of the z coordinate of the user in an ambient user interface.
Acampora et al. [18] conducted survey of the ambient intelligence in health care.In this environment an ambient intelligence can be used for autonomous and pro-active health care services, for example monitoring the health status of older adults or people with chronic diseases.Another use of ambient intelligence is assistance for individuals with physical or mental limitations.Use of the ambient intelligence in assisted living for the elderly people was analyzed by multiple researchers.Kleinberger et al. [19] created an assisted living laboratory in which they employed sensors into devices used by the elderly.Most of these devices utilized in an assisted living can be used in creating an universal ambient user interface.Nehmer et al. [20] created classification scheme for the living assistance domain (see Fig. 2).Fig. 2 Classification scheme for the living assistance domain [20] Shi et al. [21] focused on using the ambient intelligence in the computer enhanced learning.They created Smart Classroom -real-time interactive classroom with tele-education experience.Main idea was moving the user interface of a real-time tele-education system into the 3D space of an augmented classroom, so that the teacher could interact with remote students.In this classroom they used location tracking, microphone array and multimodal interaction technologies.These devices should also be utilized in creation of an ambient user interface, but since we do not currently have access to them, they remain a challenge for our future research.

Natural user interfaces
Natural user interface is similar to the ambient user interface.The difference between natural and ambient user interface is that while the user needs to anticipate and learn how to work with an ambient user interface, in natural user interface it comes naturally for him/her, as this environment is structured to behave according to his/her understanding of the world.Natural user interfaces benefit from the gestural interaction.
Norman [22] in his article describes gestural interaction in the natural user interfaces.Problem with interpreting gestures is that they do not leave any record of their path that could be backtracked.Gestures are difficult for user to discover, in contrast with GUIs in which important elements are displayed for the user and are easy to discover.Norman states that a lot of research is needed for the natural user interfaces, since the gestures are unconstrained.Metaphors of real use also need to be carefully considered, such as throwing an computer generated object could result in throwing of a physical controller.
Recognizing attention of the people in a natural user interface is particularly important since it can be used for better organization of the interface.Birnholtz et al. [2] investigate the users' awareness beyond the desktop used for face-to-face interaction through computers.They identify three behaviors that are associated with interpreting the user attention, glancing, gazing and moving closer to the point of interest.People quickly assess their environment with glancing.It shows that the person is not focused.If the person starts gazing on something, it means it gained his/her attention.Lastly when person moves closer to some object, it displays his/her interest towards the object.For this observations we recommend the use of a kinect or an eyetracker, which are unobtrusive to the user.
Birnholtz et al. [2] used peripheral-vision of system user for the the identification of attention of other participants of an online meeting.They used projector, that displays context about others' presence and display their level of interest onto wall behind computer monitor.Others' presence is indicated by avatars and their attention is observable through relative distance of their avatars towards user avatar.Ability of peripheral-vision can be used in natural user interfaces for knowing surrounding of the user.From this data withstands the need of observing glancing, gazing and physical position of user in natural user interface.

TOOLS USABLE IN AMBIENT USER INTER-FACES
In this chapter different tools that can be used in an ambient user interface are presented.Possible influence of these tools on the human-computer interaction and the arrangement of the work environment in an ambient user interface is discussed.Four tools to which we currently have access are: • Widescreen multi-touch display

Widescreen multi-touch display
Widescreen multi-touch display provides us with an ability to display any computer rendered image and allows the user to interact with it with the use of touch [23].Touch interaction is intuitive and even children are able to utilize it properly [24].
Main advantage of a multi-touch display is that it can display different interfaces, whose behavior changes, as parts of the screen could be made interactive while others stay passive.Simulating different interfaces can be used for examination of interfaces used in different domains with relatively low cost.Since the interaction is achieved through touch, it is unobtrusive to the user.Multiple difficulties come with using widescreen multitouch displays in an ambient user interfaces.The first one of this difficulties is the size of the screen and manipulation with it.When placing multi-touch display the average height of its possible users needs to be taken into consideration.The best solution to this is to use a mount with adjustable height.Through this way the height can be adjusted and different users can have access to the screen.Another aspect of the display placing is whether we put it horizontally or vertically.When positioning multiple screens next to each other in horizontal position, we get larger space for multiple users with slightly limited interaction space for everyone.It the screens are positioned vertically and put next to each other, each user has larger interaction space on the cost of the space for multiple users.Example of a mount that provides us with both options is shown in Fig. 3.

Kinect One
Kinect one is a device that tracks the user movement through observing the skeletal model of the user.It can identify basic gestures that could be used for interaction and even some hand gestures, such as thumb up or open palm.Kinect can recognize up to six people at a time, however it can track skeletal models for only two of them.The space that can be observed by kinect is shown in Fig. 4. Kinect is also able to observe the face map of multiple users and identify one of 7 different face gestures.These can be used to identify the user emotions towards different applications in the ambient user interface.Since the kinect device needs enough space to be able to observe users, this space needs to be considered in creating an ambient user interface.Since only two persons could be tracked with kinect, the applications used in said ambient user interface need to consider this fact and be adjusted for up to two persons.In case we use kinect device to track face map of multiple users we can adjust the application for more than two users, however the interaction needs to be smartly designed, such as identifying one user as the controller or only using data about users' face maps for statistic, in which none would be used for human-computer interaction.
The lower the distance of the user from object is, the higher interest the user has towards this object [2].Kinect device is able to track the distance of a user from it.This could be used in an ambient user interface to identify increased interest and what caused the rise of the users' interest.Another aspect with the kinect measuring distance of the user could be the change of displayed image.For instance, when no user is in a close proximity to the kinect under a display, the displayed text would be enlarged while the amount of text would be decreased.In case when the user would gain interest in what he sees on the screen and comes closer to the screen, the kinect would identify this and the displayed text would be changed since the user would be able to observe more from the shorter distance.
The positioning of a kinect device in an ambient user interface is however problematic, as the kinect needs to have clear view of the users.When the users are in close proximity, the kinect would not be able to track their positions and their movements.Also in some cases a calibration of the system would be needed for use of the data observed by kinect, such as when the face map of a user would be used to estimate the look of the user.Apart from calibration, the kinect device and the human-computer interaction provided by it is unobtrusive for the user.

Eye-tracker
The main goal of eye-tracker is to track the position of users' eyes and to calculate the point of regard of the user.The point of regard represents a point on screen (usually a pixel), on which the user is looking at the moment.This point can be used for interaction with the system, such as switching pages when user looks on the side or scrolling up or down in a document.The point of regard of the user also identifies the attention of the user and could be used in the evaluation of user interfaces.With this data the first element observed in an user interface can be identified and also the attention distribution over the user interface can be inspected.This knowledge can be used to further improve the user interface for computer, ambient and natural user interfaces.
With the use of a multi-user eye tracking device we can track the points of regards of multiple users.This could be used for examining multiple users' attention on the same user interface or for interactive applications.The main disadvantage for the eye-tracker however is the calibration, which means that it is better to design experiments that would last longer, as frequent calibration would lead to extensive waste of time and resources.
Same as with the kinect device, the human-computer interaction is unobtrusive to the user apart from the needed calibration and space that would be needed for the eyetracker, so it would be able to track the eyes of the user.

Wireless EEG Headset
Wireless EEG Headset is used to identify brain waves of the user.These can then be used in brain computer interface applications, in which they are used directly as an input device and the user controls the interface with his thoughts.The other use of this could be to identify whether the user is paying increased attention or if he is tired.Also several emotions and face gestures are indicated through different combinations of brain waves.
The example of an wireless EEG headset is an EPOC+ 1 , which can track 14 different brain waves channels.It is usable for 12 hours on one recharge, so it could be used for long lasting experiments.Another advantage of the EPOC+ headset is that it can precisely measure brain waves and comes with an API.
The disadvantages of an EEG headset is, that it is obtrusive to the user since the user needs to wear it upon his head.Since the sensors need better conductivity, saline based dilution needs to be put on each sensor before the headset could be used.This could cause additional discomfort to the user, because of the wet sensation on the skin.In case when an wired EEG headset is used, the wire present additional hindrance as the user could trip over it.

USING MULTIPLE TOOLS AT THE SAME TIME IN AN AMBIENT USER INTERFACE
In the previous chapter we described different tools that could be used in an ambient user interface and the ways how they could be utilized separately.However the greater challenge that could lead to increased effectiveness is to use these tools combined.These tools have some similar functions, so only the one that provides best result for a selected function should be used.
For the ambient user interface that would be adjustable for the needs of different domains and would be able to observe the user completely we can use all tools described in previous chapter.For the set-up of such interface we would ideally need space of 5 x 5 meters.In this space a widescreen touch display would be placed by a wall.Under this screen an eye-tracker and a kinect would be placed.An Wireless EEG Headset should be put with saline dilution on the side from the display.All devices would be connected to a single computer station which would execute all the calculations.
Ilustration of how an ambient user interface with a multi-touch screen, a wireless EEG headset, kinect and an eye-tracker is show on Fig. 5.When the user would come to this ambient user interface, the first device that would recognize him would be the kinect device.When the user is in a large distance from the kinect, a text should be displayed on the screen to attract his attention.When the user would approach the screen, the tested interface would be displayed on the screen.Then a notice should be shown on the screen, which would present the user with the option to start the eye-tracker calibration, such as a gesture based indication.After the calibration the user would be able to interact with the interface with his look (observed by the eye-tracker) and gestures (observed by the kinect device).Additionaly the user would be presented with the option to put on the wireless EEG headset, which would be used to observe his emotions and facial gestures or which would be used for interaction with the system.However when using multiple channels for humancomputer interaction, such as gestures, point of regard and brain waves, we need to take into consideration that these channels could indicate behavior that would be in conflict with itself.

Recommended combinations of different tools
When the user would come to the screen and would start the touch interaction, the kinect and an eye-tracker would not be able to observe the user because of the close proximity.That is why we recommend these types of use of the different tools: • An eye-tracker with a kinect device • A wireless EEG headset with a kinect device • An eye-tracker with a wireless EEG headset • An eye-tracker with a wireless EEG headset and a kinect device • A wireless EEG headset with touch interaction 4.1.1.An eye-tracker with a kinect device This combination could be used to identify the attention of the user on the interface and a small scale of user emotions through the face tracking provided by the kinect device.The interaction with the system in this case would be executed through gestures observed by the kinect device and the point of regard of the user observed by the eyetracker.The position of user in space provided by the kinect device could be used by the eye-tracker for better calculation of the point of regard, however the communication between these two devices would be difficult.

A wireless EEG headset with a kinect device
In this case two possible scenarios could be used utilized.In the first scenario, the system would be controlled with the brain waves of the user, while kinect would track the movement of the user in the user interface.These data could be used to analyze the brain-computer interfaces and the attention distribution of the user.In the second scenario the user gestures observed by the kinect device would be used for human-computer interaction while the EEG headset would provide information about user brain waves and how they changed during each gesture.This would provide important data about gesture on which the user reacts positively and those which he found cumbersome.

An eye-tracker with a wireless EEG headset
In this case the kinect would track the movement of the user in the environment and the interaction with the system would could be executed in two ways.In the first case the point of regard observed by the eye-tracker would be used for the interaction, while the EEG headset would track emotions and facial gestures of the user.In the second case brain waves of the user would be used for human-computer interactions, while the eye-tracker would track the attention of the user.

An eye-tracker with a wireless EEG headset and a kinect device
This case is the combination of the above mentioned cases.The considerable difficulty for this case however is, that each of this device has different interface, so communication between them would require through analysis and further research.

A wireless EEG headset with touch interaction
When touch interaction is used, the kinect and eyetracker would have constrained view of the user so they could not be used.However an EEG headset would still be able to track the brain waves of the user.This could be used for the evaluation of the user emotions and user experience when interacting with a user interface displayed on the touch screen.The advantage of this approach is that the user interface could be designed for any desired domain.

All devices combined
The use of each above mentioned device together requires extra caution and we do not recommend it for setting up an ambient user interface.The main reason behind this is, that it is impossible for the kinect device and the eye-tracker to provide data, while the user is executing touch interaction with the screen.However this could be avoided with positioning kinect and eye-tracker in places, from which they would have unhindered view of the user even when he would approach the screen.

Summary of the tool combinations and their use
The different combinations of the used tools have different human-computer interaction model.This is caused by the different data provided by these tools.For example, the kinect only identifies the user in distance larger than 1.5 m from the screen.The eye-tracker or a wireless EEG headset would be chosen for interaction when they would provide some data to the computer station.The eye-tracker would be used if it was calibrated and a wireless EEG headset when it would recognize brain waves of the user.
This set-up would be able to use for the research of human-computer interaction in an ambient user interface in domains, in whose the user interacts with the system while standing up.For the research of a sedimentary work environment a smaller single-user eye tracking device would be needed to be used and placed in the proximity of the computer monitor.In this case larger screen would be more beneficial, since the distance from the user could be longer and therefore there would be more space for placing a kinect device and an eye-tracker.

CONCLUSION
As the multi-display environments are becoming popular [12], there is the need of creating the objective measures for the evaluation of such environments.Preferably these measures need to be adjustable for different domains.There are some basic measures set by Vatavu and Mancas [13].These measures are designed for use with eye-tracker in a simulated workplace.The use of these measures can be redesigned for field tests, with the use of devices that can be transported from place to place.However, the displays in the field test are organized in different ways, so it needs to be taken into consideration before creating the measures.
Measure of the multi-display environments can be achieved through a special laboratory, in which all equipment is adjustable and can be adapted to the needs of the experiment.Since it is used in a laboratory set-up, it can be enhanced with the use of additional devices, such as cameras and microphones, that collect data about the experiment and its participants.Although, laboratory can never fully simulate field tests, since computer systems in real workplace could be outdated and person behave differently when under observation [8].
Firstly, an ambient user interface with motion, audio and visual sensors needs to be created.In this environment we can inspect the users' behavior and methods of evaluating such environment.The sensors in the environment need to be hidden, so users would behave naturally.Midas touch is problem connected with the use of users' gaze in humancomputer interaction.Solution by Stellmach et al. [25] aims on overcoming the Midas Touch problem with connecting the gaze controlled interaction with the touch-and-tilt device to indicate desired reaction.This approach could be utilized in an ambient interface with gestures.
There are four challenges for the future research in an ambient and natural user interfaces [18].First one is an artificial intelligence that would help the users in the user interface.Second one is design and human factors.When creating a user interface we need to take into consideration the accessibility of its elements.Third one is security, since the sensors would collect large amount of data of each person.Fourth one is the social and ethical issue, since the systems are supposed to work automatically for the user.
The main uses of different tools in an ambient user interface could be differentiated as follows.The wide-screen touch display can be used to show user interfaces for different domains while providing components with adjustable interaction.A kinect device would track the position of the user in the environment.An eye-tracker would track the attention of the user.A wireless EEG headset would track emotions and brain waves of the user.

Fig. 3
Fig. 3 Mount for a widescreen display with adjustable height and display rotation

Fig. 5
Fig. 5 Ambient user interface with: a) a multi-touch screen, b) a wireless EEG headset, c) kinect and an d) eye-tracker