Communicability Evaluation of Privacy Settings on Facebook for Android

Abstract

With the widespread popularization of Facebook, which is considered the most used social network nowadays, a concern has become constant among its users: How is users’ data privacy being assured, since one of the purposes of Facebook is actually the exposure of user data? Facebook provides privacy settings that help its users to preserve their information. However, users are not always able to access these settings easily. In order to analyze this question, two methods of evaluation were used: Semiotic Inspection Method (SIM) and Communicability Evaluation Method (CEM). The evaluation was carried out considering a very particular profile: users located in Quixadá, a small city of Ceará contryside, a state located in the northeast of Brazil who access Facebook exclusively by mobile phone. We concluded that Facebook has a real concern with its users’ information, however, in some cases, they are not receiving this message accordingly. As result, although users understand the risks of lack of privacy, they generally did not know how to use Facebook’s tools to ensure their privacy.

1 Introduction

The human being has a natural need to be social and has the impulse to seek what s/he lacks in other individuals. In order to support social interaction, social networks were born [1]. An example of a social network is Facebook, which, today, is the most widely used online social network [2]. In addition to the web version, Facebook provides a mobile one, which has versions for smartphones and tablets. Out of its 1.32 billion of users, 55% use smartphones as the main device to connect to the service [3]. The smartphone is a tool on the rise, as many users are adhering to it for ease of transportation and cost, advantages that a desktop environment does not have.

Due to the fast popularization of Facebook, issues related to security, integrity and protection of personal information have emerged primarily by users who do not feel comfortable about making their personal information available to other sites and take the risk of having external companies using them without permission [4]. Facebook enables this type of control to users through security and privacy settings.

From this delicate context, some questions can be made: Was there a decrease in privacy with the increase in popularization of Facebook? Do Facebook privacy settings adequately meet the needs of its users? Do users know how to tune privacy settings to keep track of their accounts? Are users aware of the effects of the privacy settings? Specifically for this article, we are interested in evaluating privacy settings on Facebook in relation to its communicability, which is a usage quality criterion that relates to how well the interface can communicate to the user, during the interaction, the design logic, that is, communicate the designer’s vision about who the user is, what their preferences and needs are, and what the system is and how the system can support them [5].

We then selected two HCI evaluation methods focused on communicability: Semiotic Inspection Method (SIM) [6], which, through interface inspection, aims to capture the message elaborated by the designer to be accessed by the user during the interaction with the system. In SIM, only the evaluator interacts with the system, playing the role of the user. The Communicability Evaluation Method (CEM) [6] attempts to identify communication breakdowns that eventually may result in interaction problems. In addition, to apply CEM, the user him/herself must interact with the system. The evaluator only observes this interaction.

This study was carried out with people from a region in the countryside of the state of Ceará (Brazil). The selected profile was: people with low skills in using personal computers and mobile devices, regardless of their school lever, older than eighteen years old, and that access their Facebook account from a smartphone running Android operating system.

The purpose of the research presented in this article is to identify user access difficulties regarding Facebook privacy settings. As a result of the application of the two methods of evaluation, we could observe breakdowns that were identified with both methods and others that occurred in only one of them, reinforcing the importance of combining different evaluation methods to obtain more comprehensive results. In addition to that, we could point out some suggestions for improving the Facebook interface used in the study (version 28.0.0.0.15), such as: renaming the “Synchronize photos” function to a more meaningful name, changing the order of the menu options, placing “Help Center” item as the first one, among other suggestions listed below.

The remainder of the article is organized as follows: In the next section, we talk about privacy on Facebook, presenting some research work on evaluation of the Facebook privacy settings. Subsequently, there is the methodology section, where we describe the participants profile, the scenarios that were created according to the evaluated environment and the execution of the two methods chosen: SIM and CEM. After the methodology section, we present the results section, in which we describe what was found with the execution of the two methods. Subsequently there is the discussion of results, final considerations and future work.

2 Privacy on Facebook

Facebook provides several communication and socialization functions for its users. Some of them are: (i) inviting friends who are part of Facebook to ensemble their network of friends; (ii) sending messages to them through chat; (iii) sharing news, photos or videos; (iv) posting messages and (v) commenting or “liking” friends’ posts.

In the mobile version of Facebook, there are two menus that refer to privacy settings. The first menu is named “App Settings”, where you can find settings for operations that only occur in the mobile version of Facebook, such as audible warnings when one of the user’s friends marks the user on a publication. The second menu is named “Account Settings”, which holds options for actions that may occur in both mobile and desktop versions, such as the option of analyzing photo publications, useful when a friend marks some user on a photo. When this option is enabled, the photo will go to analysis, and only after the user accepts the publication, it will appear in his/her timeline.

There is a concern with issues related to privacy of Facebook users’ information. This concern could be observed, for example, in an evaluation competition promoted in the XI Brazilian Symposium on Human Factors in Computational Systems (IHC)¹, held in 2012, which was themed after the evaluation of features offered by Facebook for data privacy control by its users. The competition sought a reflection on privacy on Facebook and the discussion of how HCI study field can contribute to the quality of interactions mediated by computational technologies.

Therefore, in this competition, several papers discussed about privacy on Facebook social network [4, 9,10,11,12]. The articles were split into use of inspection methods and observation methods. Two of them used Semiotic Inspection Method [9, 11], and other three papers [4, 9, 10] used observation methods, such as the CEM and Usability Test. Articles [4, 10] used the Heuristic Evaluation method to compare with the results obtained with observation methods. The paper [8] used data collection using online questionnaires and semi-structured interviews, using Content Analysis method to analyze the collected data. All researchers used methods to evaluate Facebook only for desktop environments.

In addition to the evaluation competition, other research works addressed issues with Facebook privacy, such as [13, 14]. The first one [13] was published in the IHC 2012 but not within the evaluation competition. This shows that it is a quite pertinent theme for the concern with the information that the users let available in the social networks. In this article, the researchers conducted a survey with 225 participants with questions related to people someone accept as a friend or even if users make selection of what they post. In the paper [14] presented in the IHC of 2013, the evaluation was basically composed of interviews, tasks and questions related to these.

All papers cited have important contributions to the subject. But the differential of our research is the target audience of the interior of Ceará state who are not technology experts and only access Facebook through smarthphones.

3 Methodology

In this section, the profile of the participants who were selected for the evaluation will be explained, as well as the scenarios that were created to perform the evaluation of Facebook for mobile phones. We also present in summary how the selected methods were executed, taking into account the environment and the target audience.

3.1 Participants

The participants’ profile included people who have a Facebook account and who use smartphones to access their account on the Android platform. In the recruitment phase, we looked for participants older than 18 years. This age limit is due to the fact that users older than 18 years can share information to the general public, to friends, to specific people, among others options. In the account created for who is under 18 years old, the option to post messages is enabled only for friends.

It was not mandatory for the participants to have completed elementary school, but they needed to be able to read, a basic requirement to fully use any online social network. We also target people who did not have advanced computer skills. Finally, we were interested in people living in the city of Quixadá and neighboring regions, where we have this specific type of audience. Our research included people who have a Facebook account and who use smartphones to access their account on Android platform, because the public in the region most of the time owned phones with this operating system. This is shown in the research performed by Damasceno [15], which may be associated with the fact that prices of devices with Android platform are cheaper than those of iOS platform. This research was conducted in 2014, in Quixadá [15] and investigated the penetration of social networks among its users. Out of the 96 interviewees, 65% did not access any type of social network, and the rest had a Facebook account. This is an interesting fact, since Facebook was founded in 2004, and, ten years later, it was known by only 34.5% of the users considered in the research. In addition, only 8% of respondents had access to the internet for more than 5 years [15], suggesting that the public in the region is composed of users with recent experience in the use of Internet and social networks.

Then, to apply SIM, the analyzers had to put themselves in place of people who only had smartphones to access their Facebook accounts, with low technology knowledge and had little experience with social networks, and for the execution of the CEM six (06) participants with different ages, schooling and time of use of Facebook were selected, as shown in Table 1.

Table 1. Profile of participants.

3.2 Scenarios

For SIM and CEM, it is necessary to create scenarios to guide the activities proposed by the methods. Scenarios describe human activities or tasks in a story that allows the exploration and discussion of contexts, needs, and requirements [16]. A scenario has a plot, which includes sequences of actions and events: what users do, what happens to them and what changes occur in the environment [5]. For the realization of both methods, we adopted the same scenarios. Three scenarios were elaborated that differed only by the task proposed in each one. In the following, we describe each of these three tasks proposed to the participants:

• 1. Enable photo sync option: When enabled, each photo taken from the user’s smartphone is automatically saved in a Facebook album. The photos remain private until the user chooses to publish them.

• 2. Disable Messenger Location: After disabling this option, when user is chatting, his/her location information will not be available.

• 3. Enable reviewing photo posts: When the user enables this option, s/he keeps track of his/her friends’ posts on his/her timeline, so no posts will appear without his/her confirmation.

An issue to note is that the first two tasks are available only in the mobile app. The configuration of the third scenario exists in both the mobile application and the desktop environment.

3.3 Execution of Semiotic Inspection Method

The Semiotic Inspection Method (SIM) is based on Semiotic Engineering [6]. SIM is useful to evaluate the communicability through inspection, aiming to build an integrated version of the metacommunication message (the message passed by the designer about the system), identifying inconsistencies and ambiguities among the various signs chosen to represent this message. It is a non-predictive method, so the evaluator explores the artifact in order to identify problems and their causes, deepening their knowledge about the artifact [17].

The inspection occurs according to the classification of the signs that make up the interface, which are divided into metalinguistic, static and dynamic. Metalinguistic signs are those used by the designer to explicitly communicate to users the meanings s/he assigned to other signs encoded in the interface and how they should be used [5], such as error messages, descriptions of interface elements and system documentation. Static signs express the state of the system and whose meaning is interpreted independently of the causal and temporal relations of the interface [18], such as icons, menus and other interface elements. And, finally, the dynamic signs are signs that express the behavior of the system, involving temporal and causal aspects of the interface. They are linked to the interaction itself and should be interpreted by reference to it, such as animations indicating system processing.

The inspection begins with the elaboration of the metamessage corresponding to each of the three classes of signs. This metamessage can be paraphrased as described in Chart 1.

Chart 1.

Model of the designer’s metamessage

With the metamessages reconstructed, the evaluator compares the results obtained to be able to elaborate a condensed version that unifies the three versions of the metamessage. Thus, in the reporting of results, it is possible to “see” the problems encountered by judging the failures of communicability.

The consolidation activity begins with the synthesis of the results, divided into possible knowledge that the user needs to have to perform the task, knowledge acquired after performing the tasks and suggestions for improving the interface to the found problems [5].

3.4 Execution of Communicability Evaluation Method

The main objective of CEM is to evaluate the quality of the communication of the designer received by the user, through the interface, in interaction time. This method increases the knowledge of designers, evaluators and researchers on how users interpret the artifact. This evaluation allows the identification of communication breakdowns that may occur during the interaction of the user with the computational artifact [17].

For the execution of CEM, it is necessary to prepare the environment where the interaction will be observed. Firstly, it was necessary to use a software that made it possible to transfer the image from the mobile phone to the computer, in order to record the user interaction. So this software facilitated the whole process of capturing the images of user interaction. There was also needed a software that recorded the computer screen to later analysis by the evaluator.

Before starting the evaluation, all users were asked to sign a consent form. In addition, participants interacted on Facebook from fictitious accounts created exclusively for the tests. We realized that the use of fictitious accounts left users more relaxed on privacy concerns because, in dealing with their own accounts, users end up being afraid of what can happen to their data. Then, it was verified that the fictitious accounts helped the evaluator to discover more communicability breakdowns by the fact that the users did not inhibit when carrying out the tasks.

For a rigorous analysis of the data generated by CEM, we performed an interview before the scenarios are executed and another one after the end of the activity. Thus, the data were crossed to elaborate the semiotic profile of the system. After the preparation of the analysis, the pilot test was carried out and we consequently updated what was necessary.

With the data collected through interview and recording of the interaction, we started with the interpretation and consolidation of the results of the CEM, which is divided into three parts: tagging, interpretation of tags and elaboration of the semiotic profile.

Tagging is used to identify communicability breakdowns, that is, moments of interaction in which the user demonstrates that he or she has not understood the designer’s metacommunication, or moments in which the user finds it difficult to express their intention to communicate in the interface. The CEM establishes thirteen tags [18], which relate these moments of difficulty or lack of understanding to possible user utterances, which represent the breakdowns that may occur during the interaction. The thirteen tags are: “Where is it?”, “What’s this?”, “What now?”, “Oops!”, “Where am I?”, “I can’t do it this way”, “What happened?”, “Looks fine to me”, “I give up”, “I can to otherwise”, “Thanks, but no, thanks” and “Help!”. For example, the tag “Oops!” is used when the user abruptly stops an action by realizing that this is not what s/he meant. The “Looks fine to me” is characterized when the user believes he finished the desired task, even when this task did not actually finish successfully.

After tagging, the tags are interpreted, through which the evaluator becomes aware of the main interaction problems [17]. If they exist, the evaluator will be able to say not only what the problems are, but also why they occurred [5]. If there is no tag associated with the interaction it means that, within what was observed, it was not possible to identify breakdowns in the communication between designer and user.

The last step of CEM is the elaboration of the semiotic profile, which consists in achieving an in-depth characterization of the user-designer metacommunication [17]. The semiotic profile is elaborated through the reconstruction of the designer’s metamessage as received by the user at the time of the interaction [5].

It is important to emphasize that SIM evaluates the emission of the metacommunication of the design, which is encoded in the interface. The MAC evaluates the quality of the reception of the metacommunication by the user [5].

4 Results

In this section, the results of the evaluation are presented. We first present whay was discovered with SIM and later the results of CEM. In SIM we can see the detailed metamessages of each task, whereas in CEM we describe the results of the three stages, tagging, interpretation and creation of the semiotic profile.

4.1 Semiotic Inspection Method

In this subsection, we describe the three metamessages generated by the evaluator, one to each class of signs (metalinguistic, static and dynamic), taking into account the scenarios that were created.

Before presenting the metamessage we will remember which are the scenarios created for the evaluation. The first was the activation of the photo synchronization option; in the second the user would have to disable the option of Messenger service locations and lastly the activation of the analysis of photo publications.

Below we reproduce the metamessage generated for the inspection of the three scenarios referring to metalinguistic signs. For better understanding, the metamessage is fragmented:

[You possible are a user who knows the term privacy, but if you do not know, you know where to look for this information on Facebook]. The application provides users with the option of “privacy shortcuts”, where there are several functions related to user privacy, such as “Who can see my stuff?”, “Who can contact me?” and “Privacy Basics”, among other options. In “Privacy Basics”, the designer, using metalinguistic signs, demonstrates his concern for the user, by advising him that he is in charge of his account and that will help him to have the experience that he wishes, offering to the user a small tutorial on privacy. The designer cares about the user’s data and tries to inform him if he does not have the necessary knowledge about the term privacy.

[You are a person who cares about your photos available on Facebook, but does not care that your friends have access to your location]. In the first task, photo synchronization is not enabled by default, besides the designer’s concern to show the user that their photos are private. When the user selects the “Sync Photos” option, which is on the “App Settings” menu, a message appears stating that the photos are private. The word “private” is bold and underlined, so that it has greater emphasis and, after synchronization is activated, the designer again informs that the photos are private. Curiously, regarding the second task, which is to disable the Messenger locale service, so that the user’s location is not told to a friend by the chat, we have seen that the location information is included by default.

[Do not worry, you have full control of the situation. In addition, you access your Facebook account by both computer and mobile]. When the user performs the task of synchronizing photos, a message appears informing the user that his photos will be available when he connects to a computer. The designer assumes with this information that his users access their accounts by smartphone and also by computer. Therefore, it uses icons from a mobile phone and a computer when the user selects the option to synchronize photos, to demonstrate how the synchronization of photos works, assuming that Facebook users also own a computer.

[You are used to using aid systems when you can not understand a certain action. To help you, we’ve set up a help center and privacy shortcuts, which explain in detail what are the basics of privacy and our data usage policy]. Regarding the first task, the designer provides a metalinguistic sign in a question mark format, in the “Sync Photos” menu, in which the user, by selecting this option, will be redirected to the help center, which is a menu created for helping Facebook users when they are in doubt about a certain action. When we inspect the second task, we do not identify any information related to Messenger locale services, only a brief message that is available when the user selects the menu for the task. Already for the third task, there are two menus in the help center to assist the user, “Publications and Markings” and “Privacy and timeline analysis”, Reinforcing the designer’s concern with photos of Facebook users.

Below we write the metamessage regarding the inspection of the static signs, following the same fragmented format with the evidence of the inspection:

[You are a user using knowledge gained from other interfaces. Knowing this, we use common everyday symbols to guide you]. The symbols chosen by the designer were a gear icon, which refers to “operation”, used to refer to settings, and a padlock icon to tell the user that his photos are private and that only he “has the key” and the power to decide when to open this padlock.

[You understand that the Messenger settings are not on the tab where your friends are available to chat, but in the application settings]. Because “Messenger” is a widely used name for chat, it is expected that this option is located in the area that shows the list of friends online in the chat. In this area, there is the same gear icon in the upper right corner. Because this static sign is also used in App settings, by association, this gear would also lead to chat settings, where the Messenger locations option would be. However, when selecting the tool, the only option that the designer made available is the possibility to activate and deactivate the chat. In this case, the designer seems not to have taken advantage of previous user experience to facilitate their interaction with the system.

And finally, below we have a reconstructed metamessage from the dynamic signs for the three tasks performed:

[You are a user who expects to sync your photos from your phone with a Facebook album, but you want your photos to be private and can only be posted when you are logged in to your desktop account]. When activating photo synchronization, the designer shows the user a message stating that the synchronized photos are private and also shows a dynamic sign in circular arrow format, which is very used to mean page update, communicating the dynamic character of this action, since when updating, if there are new photos saved in the mobile phone, they will automatically appear.

[You are accustomed to on-and-off options, so you will not have trouble making certain settings]. When you go to the “Messenger Locations” screen, there is the following information: “Location is on”. Beside it, there is a blue icon that can be turned off and on, characterizing itself as a dynamic sign, as the symbol instantly changes its color. As well as the third task, in which the user can confirm the timeline analysis, slide the ON/OFF button and the act of light (represented by the blue color) on and off, reminding to be on and off.

[You will not get immediate feedback from this setting, just when some of your friends tag you in a photo, when you can experience the effects of your actions externally]. The effects of some tasks can not be immediately perceived, because they are configurations that cause only later impacts, giving the impression of “lack of dynamic signs”. An example can be applied to the second task: When the user configures the Messenger locale service, he will only see the effect of this option when someone talks to him in the chat or vice versa, and he realizes that his location was not identified in the message. Although not immediate, this is still a dynamic sign, but perceived only “outside” the privacy setting interface.

4.2 Communicability Evaluation Method

Regarding CEM, after the evaluation with the users, the tagging stage was performed. The evaluator watched closely each video of each interaction, so that the breakdowns of communicability could be identified. Each break is associated with a tag. The graph below lists the eight tags identified in the tests, colored according to the occurrence in each task (Graph 1).

Graph 1.

Tags occurrence

It is observed in the graph that of the 13 possible tags, 8 has occurred. Among the five ones that did not generate breakdowns in any of the tasks, we highlight the tag “What’s this?”. Its absence may be related to the fact that CEM has been executed in smartphone devices, making it difficult to identify its occurrence, since the most recurrent symptom of this tag in desktop systems is to position the mouse cursor over the waiting interface signs of a tooltip about what they mean [18]. In touchscreen interfaces, there is no mouse, nor screen tips, making it difficult to identify this tag. Next, we present the interpretation of the breakdowns in each task separately.

In the first task, all users, except U4, associated the phrase “synchronize photos” to the Facebook photo album. So everyone followed the first step of looking for where the option could be to sync photos, characterizing the “Where is it?” tag. To perform the task, the preferred interaction path would be to access the “Application Settings” menu, then select the “Synchronize Photos” option, however U2 and U6 understood that synchronizing photos would be the creation of a photo album or the viewing photos of albums. Then the two users entered the corresponding menu, where you could see the “synchronize photos” (Sincronizado) option on the right side of the “Uploads” (Carregamentos) and “Albums” (Álbuns) options (Fig. 1). In this way, the two users managed to accomplish the task, however characterizing the break “Go another way”.

Fig. 1.

Syncronize tab

Users U1, U2, U5 and U6, when creating an album, thought that they had completed the task, when, in fact, they did not, characterizing the tag “Looks fine to me”. For this task, U4 was the only one who did not ask the evaluator for help. All the others asked for help, characterizing the tag “Help!”, because they could not understand where the function to synchronize photos would be located.

In the second task, users U1, U2, U4 and U5 completed the task in a few steps. U3 and U6 users went different ways. User U3 performed a sequence of steps within menus, and in the end realized that he was not on the right path and returned to the starting point, characterizing the tag “I can’t do it this way” User U6 has associated the word “Messenger” with the chat window that shows the list of friends who are online and offline, choosing the configuration gear sign. The subsequent screen, only with one option to activate and deactivate the char surprised the user, characterizing the “What happened?” tag, as shown in Fig. 2.

Fig. 2.

Chat tab

In task 3, U1, U2, U3, U4 and U5 performed the same steps that characterized the same breakdowns: first the “Where is it?” tag, then the “I can to otherwise” tag. They entered the application settings menu, thinking that the solution to the task would be in this menu and then selected the “Photo tagging (alerts enabled)” (Marcações de foto (alerta ativado)), option, which means that when you mark a photo, the user will be warned by an audible alert, thus characterizing the break “Looks fine to me”, because the users thought that they had carried out the task with success. The user U1, after this action, chose in the same menu the option “Publications in the mural (alert activated)” (Publicações no mural (alerta ativado)), occurring for the second time the “Looks fine to me” tag. These options can be viewed in Fig. 3. After this sequence of steps, users U2, U3, U4 and U5 have requested help, characterizing the tag “Help!”.

Fig. 3.

Application configuration menu

Data collected from the interview

Although CEM did not establish the presentation of interview data with participants as one of its steps, we decided to present some relevant passages in this section as the interviews revealed important information about how they view the privacy issue on Facebook.

One of the interview questions was “How do you understand privacy on Facebook?”. U2 said that he uses Facebook as a means of communication between him and his son who lives in another city. But when asked about what he understood about privacy on Facebook, his answer was with another question: “Like, for everything I do on Facebook, any other person who is my friend is able to see?” After an affirmative gesture from the evaluator, he went on to say that “This is not cool!”, realizing the risk that he himself should be putting on his profile on the internet.

U3, when asked the same question, was much more detailed in reporting that he did not lose privacy by joining Facebook because: “I always thought that, not that it protected my privacy, but I never went any further, because I only have on Facebook [as a friend] whom I know [offline]”. U3 has been unconcerned about privacy issues because it only has close people connected to it through the network. He told a case in which he even excluded some people he knew, but whom he had no close contact with and therefore did not qualify as friends.

U4, when asked if it ever used a privacy configuration tool, replied: “Actually I try to put as few things as possible, so I do not need to use any privacy tools, just put what I believe other people can see.” This suggests possible users’ difficulties in not knowing how to configure their Facebook, which is why they prefer to police themselves, for fear of exposing their information.

U5, when asked if she had ever been in a situation involving her privacy, answered yes: “I had to share a video of my daughter just for her boyfriend and I ended up sharing it for everybody, because I did not I knew how to put it [only] to him.” This user attempted to perform an action that should be simple and, for inexperience, the opposite happened. For this reason, the concern is real whether users can configure their privacy preferences by using Facebook.

At the end of the interaction of the six users, they were asked if they had ever performed these settings, only U4 had did one of the settings, which was the one for photo analysis, but explained that it was a long time ago and that he no longer remembered how to perform the action. When asked about suggestions for improvements, two of the six users, U1 and U3, suggested changing the name “synchronize” because they did not know what it meant.

U1, when asked about what he had learned about privacy after the test, responded: “I thought everything was exposed, I did not know I had those options. [When I] have more time, I’ll look to learn the other options.” U6 said, “I learned that I can further secure my privacy,” when asked about the same question.

U5 was very happy to learn how to set up photo analysis, whose opinion was: “I learned to analyze [the photo tags], I did not know I could, for example, if the person tagged you in a photo and you did not like the photo, so you can block your friends from seeing this picture, citing one example: “I went to the [company name] meeting once, and they took a picture, I was too fat, it was horrible, and this photo everyone saw because [I] did not know that it was not to put on my Facebook.”

With all this information, we could see that users care about their privacy, as they cited cases of privacy risk, because they did not know how to configure Facebook to the best of their ability. In the next section, we present the semiotic profile, which is the last stage of CEM.

Semiotic Profile

The semiotic profile has the objective of identifying and explaining the failures in the system’s communicability. As in SIM, we used as a template the message that is set out in Chart 1. Below the metamessage was reconstructed according to the evaluator’s perspective from what was observed in the users’ interactions with the system, always aiming to tell what the designer wanted to pass on to its users.

You are a user who cares about privacy, and we therefore make various features available that you can enjoy. We know that you use Facebook extensively, but you do not know that there are such privacy settings, and if you know they exist, you do not know where to find them. By not knowing how these settings work, you feel fear and insecurity about your data that is available in your Facebook account. By not knowing about these features, you choose not to look for the information, for fear that something bad happen with your data. The fact that Facebook handles personal information ends up leaving you distressed with every action you take, for fear that your data will be put at risk, weakening or extinguishing any desire to seek or test the functions that Facebook makes available to you.

5 Discussion of Results

With the completion of the two methods and the consolidation of the results in both, we could see that both CEM and SIM methods pointed out some breakdowns in the interface communication, both in sending and receiving the messages.

Regarding the first task, with CEM, it was seen that users did not know the function of the “synchronize photos” option, even asking the evaluator for help. But with SIM, it has been seen that there are various forms of help that support users who do not know how to perform certain actions. In a way, this help is not reaching the users to whom this research was directed. In CEM, it was realized that the users’ greatest difficulty was to understand what the word “synchronize” meant, making them rely only on the word “photo” of “synchronize photos”, to seek resolution of the task, and to focus on the options for photo albums. However, they ended up looking in the wrong places, difficulty that was not identified during the semiotic inspection.

For the second task, five out of the six CEM users performed the task with ease, since the path to solve this task was the same as the first task. Using this knowledge, they ran the scenario correctly, but one of the users attempted to access the chat window, which is understandable, because the evaluator, when executing the SIM in the second task, also noticed that, in this window, there is a static sign (the gear) that refers to a tool that Facebook itself uses in the settings menus.

In addition, the evaluator also noticed in SIM that the message that is displayed to the user when he selects the Messenger location service option is somewhat confusing because the designer reported that it was necessary to select a symbol before the message was sent. This message ends up confusing the user in the evaluator’s view. However, CEM showed that, even with the confusing message, the user was able to perform the task without major difficulties, most likely due to previous experiences with similar on/off options.

In the third task, five out of the six users performed the same sequence of steps on CEM. As the first two tasks were performed in the application settings menu, users also sought from this menu the resolution of the third task. In this menu there is a Photo Tagging option, but this option is only to set audible alert, confusing the users.

In SIM, the evaluator discovered that the designer used the metalinguistic features to explain and guide the user so that he could protect his privacy, such as privacy shortcuts, in which there is a specific menu for notions of privacy. Facebook also has a policy that describes how it devotes itself to take care of its users’ information. With SIM, Facebook’s concern with user data was very evident, but with CEM it was possible to discover that this information, in a way, is not reaching the user in the way the designer intended.

5.1 Suggestions for Improvements

After discovering which breakdowns were found by the two methods, some improvements can be listed so that in the future these failures do not happen.

• ➓ Change the name “synchronize” to a word that is closer to the user’s vocabulary, such as sharing photos from the phone in an album, because users did not understand what the word meant. Another suggestion is to use metalinguistic signs to explain what the static sign means, a strategy that Facebook itself adopts in other actions.

• ➓ Put the “Help Center” menu above the “Settings” menu, so this option would be more visible to the user. With this modification, they would possibly access this menu first, before executing the settings.

• ➓ In the second task, a targeted help is required for the Messenger locale option, as it is not clear from the user message how to select this option.

• ➓ Another suggestion would be to remove the blue arrow, which is located on the “Messenger Service and Location Options” screen, for the reason users do not understand that the blue arrow will appear when the user opens a chat window only, ultimately confusing the user. Since there are users who only use Facebook through the mobile application. This sign ends up confusing the user, since he will not deal with it in the interface he uses.

• ➓ In the third task, it would be of great help not to leave the user’s easy access to the option of sound notifications for markings of photos, since the user was confused with this option. If it was placed within a specific menu of sound notifications, it is possible that the user did not enter the menu, thinking that it is in the desired option.

The Facebook application for smartphones was recently upgraded to version 108.0.0.17.68 (January 2017), and even without detailed inspection, it was already possible to identify some differences, such as the “Synchronize photos” option, which was replaced by a standalone application, similar to Messenger, titled Moments. This first observation is evidence that some of the problems identified were revealed to be real by the Facebook team, to the point of being resolved in later versions than the one used in our evaluation.

6 Final Considerations

In this research, we performed an evaluation of the communicability on Facebook’s privacy settings, using two methods of Semiotic Engineering: SIM and CEM. It is important to notice that the two methods were chosen to be applied together, in order to have a greater relevance in the results. Some ruptures found in CEM were also confirmed in SIM, but there were also a few cases in which the evaluator did not find any difficulty in performing the scenarios during the inspection, but in CEM users experienced difficulties, even asking the evaluator for help. There were also situations where the evaluator identified problems in SIM that did not materialize in CEM.

At the end of this research, it was seen with SIM that Facebook cares about users’ data and that it offers a series of options that allow the user to secure their information according to their will. However, in many respects, the CEM showed just the opposite: users generally did not know that these options existed and therefore often experienced situations of lack of privacy with their information, as reported during the interview.

Therefore, it is expected that, with the results and the ruptures found, we might have contributed to future versions of Facebook interface, so that, in addition to put effort to produce a pleasing interface in aesthetic terms, they also improve the communicability concerning privacy issues, so that users can use Facebook the best and safest way.

In addition, we also see as contribution of our research, a complete example of joint application of the SIM and the CEM and also the reflection on privacy in virtual social networks.

We identify as an opportunity for future work, an analysis of the same scenarios performed in this article, as the new application update. Another option would be, after this analysis of the new version, compare the results of the two evaluations and see what has been improved. Another possible deployment, would be after discovering the ruptures of Facebook settings, use Codesign or Participatory design to analyze whether the ruptures found would be eliminated or not if the user had participated in the design. It would also be interesting to apply a similar research focused on other social networks.