An Investigation of a New Social Networks Contact Suggestion Based on Face Recognition Algorithm

Automated comparison of faces in the photographs is a well established discipline. The main aim of this paper is to describe an approach whereby face recognition can be used in suggestion of a new contacts. The new contact suggestion is a common technique used across all main social networks. Our approach uses a freely available face comparison called ‘Betaface’ together with our automated processig of the user’s Facebook profile. The research’s main point of interest is the comparison of friend’s facial images in a social network itself, how to process such a great amount of photos and what additional sources of data should be used. In this approach we used our automated processing algorithm Betaface in the social network Facebook and for the additional data, the Flickr social network was used. The results and their quality are discussed at the end.


Introduction and State of the Art
There are uncountable pictures on the Internet nowadays, additionally, a lot of them can contain information about relationships between the people.The very first question is how the information can be extracted and where it can be used after the extraction.
Social networks seem to be the ideal place where extracted information about relationships can be used.
The following text focuses exactly on this connection between the analysis of the faces in photos and the usage of this data in the social network during the new contact suggestion process.The process of new contact suggestions is currently based only on text data; such as place of birth of a user, his or her favorite topics in magazines or on the Internet, and so on.It is obvious that there is also a number of problems that must be resolved, especially those related to reliability of facerecognize algorithms.
Amongst the many systems that exist for face recognition we count for example following.The part of the project FaceBots: Social Robots utilizing Facebook [20], called Friends with faces [21] is very close to the subject of our research.The goal of whole project was to develop a robot with face-recognition ability and dialog capabilities that is connected into the social database.With these features the robot is able to analyze information in the social network and use it in the real world.The face recognition could be performed on real world photos captured by robot's cameras or photos downloaded from Facebook.After the analysis the robot can 'decide' who is friend of who.
Another way to find possible friends is to monitor the user, as described in [24] and collect important data which is then used for new contact suggestions.The work [11] describes the new face recognition method suitable for social networks.The face "hallucination" method should solve the problem of poor quality photos that are often available on public social networks by a patch-based learning method in discrete cosine transformation and by generating more refined highresolution face images.In the [12] the authors rec-ognize faces by the method based on graph theories.A face is compared by an external reference face graph, the identification is done according to centrality measures.
In our approach we would like to replace the analysis of social context by the analysis of commonly available photos which we can either match together or not and use it as a tool for any other purpose in this field.The proposed method can also be used as a supporting tool in suggestion/search tasks, e.g. in [19].
The software used in this research was Betaface and has been tested for its capability to recognize faces and its attributes in order to use such SW in our further research on social networks.
Our results confirmed that face recognition techniques could be used as a supporting tool during the process of new contact suggestions.The amount of test data was not so large, due to use of the real data and for the evaluation process, full access to the tested profiles was necessary.This resulted in usage of our own profiles on the social networks.The results also show that even if the face recognition phase concluded with an incorrect evaluation of the person, an inappropriate new contact suggestion was not provided to the user because of the additional text data evaluation; more details in Section 6.

New Techniques for Contacts Suggestion
Most of the currently well-known social networks use techniques that are based on processing only text information in new contact suggestions.Samples of those most popular techniques for new contacts suggestion are listed in following section.The process of finding new contacts is a perfect area where a face-recognition element can be used.Today's most popular techniques can be modified so that the processing of textual information can be enriched with a subprocess of facial analysis usually present in the photos on social networks.Lets mention a few of them: • New suggestions according to common friends is one of the simplest techniques based on the premise so if a high number of your current friends (contacts) have connections to the same person, you will probably know the person, too.
• New suggestions based on similar free-time activities and hobbies.In most of the social networks, there is the possibility to create groups of users for example with common interests.If you are involved in one of these groups there is a possibility that you will probably know some of the other users from the same group in real-life which means that they are good candidates for your new contact suggestions.This technique is usually combined with other techniques for better results (e.g. with previous technique).
• New suggestions based on same or similar occupation/jobs department.In the case of similar occupations, the additional time period filter can be applied (search only for people who worked in same company in same time period).
• Import of contacts from external applications is quite delicate topic, especially in Facebook.The user of the social network has the possibility to import data of the other persons, for example from his e-mail client.Then the social network tries to find the imported people inside its database and the new connection within the network can be provided.The problematic issue is that the social network can store the data about the imported people and use them for future advertisement without the explicit permission of the people that were imported from the external email-client.
• New suggestions with the help of external information -in this situation collected data of a previous type can be used, the typical scenario is: User A uploaded contact list CL A .User B also uploaded CL B .The connection between user A and user B is not yet established.There is a subset M of the all users that uploaded their own contact lists (includes users A and B); the social network chooses users from subset M , where there is a higher probability that they have common subjects of interests (or similar occupations or place of residence) and then the contact lists of the users are analyzed.
For example let's choose users A and B. If the number of common contacts in contact lists CL A and CL B exceeds the threshold that had been set before the analysis began (the threshold is usually computed based on percentile number of all users in contact lists), new contact suggestions between users A and B are realized.This technique is very similar to the first one, but with the benefit, that people from analyzed contact lists do not have to be users of the social network.
• New contact suggestions based on interaction with a user -when a new account is created, the user typically has only a small number of contacts so the previous techniques would not be effective.If the user has no new contacts for a long time period after the first login, the social network can automatically ask his current contacts for help with creating new contacts for this user.
This function mainly prevents the user from leaving the social network because the network is not attractive to him/her due to his/her limited number of friends there.
• New contacts suggestions based on analysis of the user's interaction on web -the existence of this method is not proven by official reports from the social networks but there is a high probability that the most of the well-known social networks use some kind of user-tracking algorithms.The analysis of the user's behavior can be based on information about profiles on the network that user visited (or visited more than once), or your favorite search phrases.It is important that all of the analysis techniques meet license agreements between the user and the social network usually in agreements there is a paragraph where the users agree that the social network can collect data about their behavior for future improvement of the services provided by the social network (data will not be provided to any other web sites).
• More sophisticated methods for new contact suggestions -Facebook launched a new service called Graph Search [5] (which currently works only with the English language).The service provides a new way for users to search through the social network (Graph Search uses phrases from common human communication).The service is based on the composition of common search phrases, e.g.Search for people who love rock and live nearby.The algorithms in Graph Search can be used in the future for new contact suggestions with similar hobbies as the user.

Face Recognition Techniques
Nowadays a specific primary objects of the face are used for face recognition in most of the current algorithms [26].The primary objects are for example eyes, the nose or ears.When these objects are found in a face the algorithms usually compare the positions and sizes of the objects and then this data is used for face comparison.Many problems, such as obstruction of the part of the face, can occur.Another problem that the algorithms have to deal with is the fact that the size and distance between primary objects may vary during different emotions of the examined object (see Fig. 1).Success in dealing with these problems depends on how each algorithm can adapt in less than ideal conditions.

Basic Classification of the Algorithms
Algorithms for face comparison can be divided to the following groups (depends on how adaptive they are).

1) Semiautomatic Algorithms
These algorithms represent the easiest form of face recognition.Interaction with the user is required before the recognition can start.The user has to mark primary objects in the face or mark the approximate area of occurrence if there are some of these objects missing (see Fig. 2).After this phase, computation on the primary objects begins [2].W.W. Bledsoe [1] was the first to try to implement this system.The whole system was then improved in Bell Labs where the vector composition of the primary objects was determined.The improvement leads to better results during the analysis phase [10].Algorithms based on these systems are still used but for our purposes they are not suitable since user interaction during the face recognition needs to be eliminated.

2) Automatic Algorithms
Similar comparison techniques to the previous group are essentially also used in the automatic algorithms.
Only the manual marking of the primary objects is replaced by an automatic detection of these objects.This automatic detection was first implemented by Fichler and Elschlager [7].Their algorithm was then improved and worked on with a parametric model of human face [27] where the primary objects were detected more reliably since the parameters of this model were determined for each photo individually.The level of errors during the comparison of faces is also unsatisfactory in this group of algorithms.Especially if the picture of the face is taken in bad conditions (for example: some parts of face are missing or they are covered by shadows).

3) Self-Learning Algorithms
The algorithms which are able to self-learn are the best option for our purposes because the all above mentioned problems can be solved by them.Problems with emotions in a face can also be solved.Associative networks which implemented basic self-learning algorithms [13] were used in the beginning.This was then improved by using recursive results propagation.When neural networks began to be used the algorithms become more effective.The results of self-learning are shown in Fig. 3.

Requirements on Algorithms for New Contact Suggestions and the Best Candidates for Purposes of our Work
Firstly, the basic requirements for the algorithm that will be used in our solution must be established.One of the most important aspects was that the service must be freeware.The most important requirements are as follow: • To be able to process a large amount of the photos in a short time period (without any significant limitation due to free access).
• The short-term storage for collection of the analyzed photos (the short time period should be at least in minutes).
• To provide additional information in a text form so the analysis of the faces can be done in cooperation with some type of pre-filtering based on text info ( for example text info can contain predicted gender).
• To be freeware or at least it can be payware with a special license for academic usage, as for example Luxand FaceSDK or Betaface is.

1) Luxand FaceSDK
Luxand is not provided as full-service ready for use but it provides SDK that enables its functionality for face recognition.Their algorithms detect and use 66 special points (equivalent of the primary objects mentioned above, see Fig. 4).Analysis of the face based on this special points is then performed.There is a freeware test application that can be used via web page, the complete SDK is available only as a payware service.There is also no additional information about the algorithms that are used for face recognition.The price of the whole SDK is not fixed but it is calculated based of the specific usage of the SDK.The advantage of the service is that it is a well-known application (it is also used for login services into different systems [18]) with very good support.There is also the assumption that the algorithms for face recognition will be improved in the future.
The disadvantage is that it is not freeware service and the sufficient limitations of the test application available for free are not sufficient for our use.The result is that the application will not be used in our solution.

2) Betaface
Betaface is fully freeware service.Basic communication is done via classic web interface, where a Silverlight ap-plication runs.The results are provided in text form, with supplementary graphical data.Betaface also provides an API through which the whole communication can be carried out (via standard GET and POST methods).Information about the internal Betaface algorithms is not public, however, from the provided data we can determine that one of the algorithms based on analysis of the primary objects is used.
The main disadvantage of Betaface is the fact that if a large load of images is uploaded in a short time period, an artificial slowdown of the analysis process can occur (but this slowdown is not significant).The artificial slowdown disappears as soon as a license for commercial purposes is used.
The advantages of this service are that the algorithms are robust enough and it is fully free-ware.It was decided that for our solution, Betaface will be used.Notwithstanding the fact that the service for face recognition can be at anytime, replaced by a better one.

The Experiment Design
In this paragraph we briefly describe what kind of social network and software recognition has been selected and used.

Social Network Selection
A very important aspect of the whole experiment is the correct selection of the social network itself.The social network with a sufficient number of users and with a well-maintained API should be selected.The main subject of interest of the selected social network must be the socialization of its users.It means that networks like Youtube or Flickr are not ideal for our purposes.
The database [14] of the well known social network (where connections between users is enabled) was used during our selection process (containing such networks like Facebook, LinkedIn, MySpace amongst the others).The database is part of the project [9].The cardinality of users is based on DAU number.The DAU is equal to the number of the active users per one day.In the database, there is also information, if the special framework for robotic search through connections between the users is supported in specific social network (FOAF output).

Betaface Algorithms
In our solution, Betaface service is taken as a black box.However a basic test of its algorithm was per-formed, mainly because of the possibility of replacing the algorithm with a better one in the future.Betaface says [15] that its algorithm reaches 90 % ID rate at 1 % FA rate for a single face per person in its internal tests.These internal tests are comparable to the LFW [16] test protocols.
The dataset of images with faces provided by Ralph Gross from Carnegie Mellon University was used for our internal test.This set of the pictures was taken over two months when 63 people were photographed and more than 4000 images [23] were created.Three random persons (two men and one woman, see Fig. 5) were chosen for our test.Images of those selected persons was created in different lights and angles conditions and consequently analyzed.The example of different conditions is shown in Fig. 6.A more accurate description of the categories of pictures is listed below:

Categories of the test photos:
• 1: Reference picture of a person.
• 2-4: Face is located in front of the camera.The light varies.
• 5-8: The angle between face and camera is bigger (horizontal and vertical).artificial light.
• 9-11: The picture of face is taken from the side.Normal light conditions.
• 12-14: Face is located at a normal angle.The light varies.Table 1 shows particular results of Betaface's face recognition capabilities.The x marks the situation where the face was not detected in the picture despite the fact that the face was detectable.The final result of this test is shown in Tab. 2. The problems occurred mainly during the analysis of pictures taken with large angles between the face and camera.In most of these situations the faces were captured almost from the side and the algorithm could not have found critical points in the faces (see Fig. 2).The susceptibility of the algorithms to different light conditions has not been proven.The results for persons A and B were basically the same.The only difference appeared during the analysis of the person C. It was caused by the lower positioning of person C. The face of the C was located in the lower part of the image which caused lower resistance to larger angles between face and camera.The result shows that the average success rate of Betaface service was 58 % during face recognition.

External Service's Communication
The communication layers with the external Betaface service and the social network are implemented inside our new application without the use of external libraries because it allows better control over the various phases of communication, and less data traffic in most situations.The only exception was Flickr, where an external communication library was used because the authentication phase in this service is quite specific.

1) Betaface Communication
Basic communication is done through XML documents (see Fig. 7).Betaface authors also offer a library for the .NET Framework but it is needlessly complex for our purposes.For this reason we decided to implement our own communication layer.
Beta service Beta service [2] xml with id of received image [1] send xml that contains image data [3] xml with request for info about face (can be sent n-times) [4] xml with requested data (can be sent n-times, when data not ready=empty xml) processing of data Fig. 7: Data transfer during communication with Betaface.
Each picture has to be encoded to base64Binary and then serialized in the data stream.For this purpose our own BetafaceImage class was implemented.

2) Facebook Communication
Facebook's developer pages give a full description of all communications methods.All queries could be transfered as standard GET and POST methods, moreover they can be done via HTTPS with SSL, see Tab. 3. Data format is mostly JSON.A Facebook user is identified by a unique ID, since the user's name could be ambiguous in case of common names.Typical response from Facebook when the query for the user's details is sent to its API.Specific token allows us to access more details about the user, however, the user's permission is necessary when detailed information is required.
Each external application needs to be registered in Facebook which allows communication and data analysis.
The analysis process of existing friends is shown in Fig. 8. Once the friends list of a particular user is obtained, the details about each friend (such as name and profile picture) must be requested separately.Then a pairing of profile picture and the data of the specific person follows and if the face is recognized, the entire record (profile picture with the face and name of the person) is stored for the later processing.Friends that cannot be recognized by their faces are avoided in this way.

3) Flickr Communication
This is the only case when the external library for communication [17] is used.Each external third-party application that wants to communicate with Flickr must be also registered on Flickr's web site, similarly to the Facebook case.After the registration process a unique ID and secret key are assigned to the application.With this data the application can then communicate with the Flickr network.
Flickr does not support the automatic logging of users in third-party applications due to security reasons.In our application it is not possible because the user filled login information into the service only once and then it is used automatically.It is necessary that during each new access to the network, when the login is required, the user must be re-authorized through a Web interface.In our application .NET WebBrowser object was used so we could integrate the Internet browser with our application.This avoids opening an external web browser that would otherwise be used.The specific form contains an event that is invoked when a web browser is closing.The method wvAu-thorizationCompleted is connected to this event.A final check if the user gives to our application requested permissions is done in GetPermission method.
Even during communication with Flickr, our own auxiliary class for communication is implemented.

4.
Case Study: Verification on Facebook's Friends and Flickr Pictures Flickr and Facebook were selected as a real world case study.The author's own accounts were used.The Facebook account contains 77 friends in total, 75 of them could be possible to track.Two of the friends have private settings set to the highest level and they cannot be analyzed.
The principle of the new contacts suggestions can be divided into two parts.In the first part the initial analysis of all current contacts in the social network is done.The result of this part is a database of current contacts in the specific social network (includes data about faces of the users).
After the initial analysis, search for new contacts can begin.Figure 9 describes the whole process of the search for new contacts.It is obvious that in this stage, the database from the previous part of analysis is used.
The final result is a database of possible new contacts that will be provided to the user of the social network.
After an initial filtering, where the friends without detected faces on the profile photos were removed, 51 users were left for further testing.In Flickr, a series of 20 photos that capture the user of our service with other people was created.The entire series was divided into two parts, 10 photos in each one.In the first, there were only photos which contain the names of the persons who appear on them in its tag information.Within these ten, five photos were with the current Facebook friends of our user.The remaining five photos contained faces of strangers.The second part of the series includes only those photos without names in the photo's tags.Again, five of them were photographs with existing friends on Facebook and five with previously unknown people.This whole series of The names of the people in the photo's tags were in a format that allowed us to find the people on the social network.If the image contains an unknown person(s), it is possibly a new contact and additional text data is available; the column new faces contains the count of these possible new contact(s).If there is an asterisk at the number, it indicates that the person cannot be found (tag photo does not name these people).An example of one photo that contains a new person with information about his name is shown in Fig. 10.One new friend was suggested to a user of our application during the analysis of this photo.

Future Work
Above mentioned methods are amongst many.Beside those algorithms, an open area of possible research is also in the field of artificial intelligence.Research papers on face recognition based on machine learning, fuzzy logic or neural networks are reported in papers like [28], [29], [30], [31], [32] and [33].Nice overview of contemporary methods is done in [34].Based on existing methods mentioned above and status quo on the field of artificial intelligence and softcomputing, it is clear that further research shall be focused on hybridization of such methods.It is clear that, for example, use of neural networks designed and learned by evolutionary algorithms or/and fuzzy sets number and its position on fuzzy universe as well as base of rules or inference rules set up by evolution is promising area of future research, in which we would like to continue.

Conclusion
The main goal of our work was to find a way how face recognition capabilities can be used as a supporting tool in social networks, based on existing algorithms and services.The process of new contact suggestions was chosen as an ideal area for the usage of face recognition.The sub-task of the work was to create an experimental test algorithm and application that shows the capabilities of our solution.
The external source of images -Flickr and the Facebook social network were used in our practical experiment.We established the test set of 20 photos where a total of ten people could be recognized and suggested as new contacts for the user of our application.
The Betaface algorithm successfully recognized all faces in the whole set of photos.All incorrect recognitions of faces (algorithm detected a face where no face was actually presented) were also presented in the results.The final result was not affected by these incorrect recognitions.A total number of 4 new people was marked as potentially new contacts for the user.This means that faces of these people were also recognized in profile pictures in Facebook.
No faulty designations of a new connection were made, which is a satisfactory result of our experiments.This is ensured by a two-level confirmation of possible new contacts suggestions.Firstly, the new contact is found with help of text information (name of the person) in the social network.Then the profile picture of the new contact is compared with the face from the photo.The result confirmed that this approach is robust enough for our purpose.
A higher number of errors occurred in the phase of new tags creation.This process is based on the information about the already existing friends in the social network.In seven photos at least one error appeared in the text information that was added by our algorithm.This level of error is directly proportional to the quality of the face recognition algorithms provided by the Black box (Betaface in our case) and can be improved by using a better application for recognition in the future.Since this feature for adding tags to the photos was only complementary to the feature of the creation of new contact suggestions, our experimental approach can be regarded as a successful one.

Fig. 1 :
Fig. 1: Changes of the primary objects during different emotions of the man [25].

• 15 - 18 :
Face is located at a normal angle.Open and closed eyes.Person A and B are captured with and without glasses.Common light conditions.c 2016 ADVANCES IN ELECTRICAL AND ELECTRONIC ENGINEERING

Fig. 9 : 4 :
Fig. 9: Own algorithm of the analysis of possible new contacts.
Tab. 1: Betaface face recognition capabilities in different situations.