A survey on automatic techniques for enhancement and analysis of digital photography

Abstract

The fast growth in the consumer digital photography industry during the past decade has led to the acquisition and storage of large personal and public digital collections containing photos with different quality levels and redundancy, among other aspects. This naturally increased the difficulty in selecting or modifying those photos. Within the above context, this survey focuses on systematically reviewing the state-of-art on techniques for the enhancement and analysis of digital photos. Nevertheless, it is not within the scope of this survey to review image quality metrics for evaluating degradation due to compression, digital sensor noise, and affine issues. Assuming the photos have good quality in those aspects, this review is centered on techniques that might be useful to automate the task of selecting photos from large collections or to enhance the visual aspect of imperfect photos by using some perceptual measure.

1 Introduction

In the late 1990s, there was an immense growth in the digital photography industry. Manufacturers began to produce digital cameras on a large scale and at decreasing prices [119]. Great changes have been noticed in photographic technology and practice since then. When using consumer analog film, the number of photos was limited by the roll size (which usually allowed at most 36 photos). Nowadays, with large capacity re-writable memory cards (e.g., 256 GB), the number of photos that can be acquired/stored has increased by approximately three orders of magnitude (if considering digital images, captured with a resolution of 8 MP). Digital photography also changed the way photos were printed. With film, photos had to be developed first, in order to be seen, whereas when shooting with a digital camera, printing is no longer a requirement, once it is possible to preview images in the camera viewer or on a monitor screen, and then to decide which ones to print.

One consequence of those changes is that taking photos has become an almost costless task. Thus, the judgment of what could be a good shot and the care for adjusting camera settings for a specific scene becomes less usual for most consumers and even for some professional photographers. As a result, large amounts of photos are taken and stored daily. This causes difficulties in selecting which ones to print or to publish, e.g., in digital albums. In summary, this results in a scenario involving a large amount of stored photos from which just a small part will be printed. In this survey, consumer photos are considered the ones obtained (1) with minor adjustments in camera settings, (2) aiming at portraying daily events, and (3) barely exploring art basic techniques. On the other hand, professional photos differ from consumer ones by the use of more elaborate techniques and better equipment utilization, which might improve the photo quality. In this survey, professional photos are not necessarily obtained by a professional photographer, and do not encompass other connotations of professional photos (e.g., artistic or journalistic).

There are several recent applications for which photo processing is an essential intermediate task, e.g., photo collage [149], slide showing [44], browsing [68, 85, 112, 150], storytelling [53], and photo summarization [28, 110, 117, 141].

The algorithms reviewed in this survey are organized into two categories: enhancement and analysis. Each category is divided in other sub-categories. An illustration for the division that is used in this work is shown in Fig. 1.

Fig. 1

Block diagram illustrating the sub-areas in which this work is subdivided

While enhancement algorithms are intended to modify the image in such a way that it might become better-looking or appealing, analysis algorithms are designed to assess photos according to some criteria, such as composition, aesthetics, or overall quality.

A number of papers have been written on both image enhancement and analysis in the past years. This survey focuses in work on image processing techniques that were already tested or may be directly used in specific problems of photo enhancement and analysis. In order to avoid ambiguities, in this survey the words photo and photography are strictly related to consumer and professional photography.

Image enhancement algorithms can be classified as on the fly and off-line. While on the fly algorithms modify the photo conditions before the photo is taken, off-line algorithms perform changes after the acquisition took place. Although the on the fly algorithms might lead to better results than off-line algorithms, they must run faster since it might be necessary to do this in a real-time operation. Off-line algorithms are limited in the sense that they do not allow scene changes, e.g., it is not possible to zoom out from a photo or ask someone to open his/her eyes. However, there is no a priori time frame to produce the enhancement result.

Image analysis algorithms can be classified as assessment, information extraction, and grouping algorithms. Assessment algorithms analyze the visual aspect of a photo in two main facets: aesthetics and with respect to the image quality assessment (IQA). Formally, the main goal of IQA algorithms is to predict ratings in a human-like manner [21]. Although this definition is very broad, the term IQA is typically used to denote the evaluation of the image degradation (e.g., due to lossy compression or noise) [21, 62, 63, 118, 167, 178]. Therefore, in this survey, IQA is used with this latter meaning. There is also some ambiguity regarding the use of the expression aesthetics quality assessment. In this survey, aesthetics quality assessment algorithms are defined as the ones whose goal is to assign a score (or a class of scores, such as professional and amateur) to a photo based in the analyzed feature, e.g., photographic composition rules or number of faces found, as used by other authors [7, 74, 126, 145]. Moreover, information extraction algorithms search for elements of interest, such as the place a photo was taken, the existence of faces, and the presence of specific people in the environment, among others. Finally, grouping algorithms are defined in this survey as the ones which analyze images in order to find similarities between them.

It is not within the scope of this survey to review image quality assessment for evaluating degradation due to compression, digital sensor noise, and affine issues. Assuming the photos have good quality in those aspects, this review focuses on techniques that might be useful to automate the task of photo selection from large collections or to enhance the visual aspect of imperfect photos by using a perceptual measure.

This survey is organized as follows: in Sect. 2, the methodology employed for finding related work is presented. In Sect. 3, the work on image enhancement is reviewed, in particular, enhancement that could be performed to increase the quality of a photo in a printing or selection scenario. In Sect. 4, work on image (and photo) analysis are reviewed. In Sect. 5, the main issues found in the reviewed approaches are discussed and summarized. Finally, in Sect. 6, some conclusions are given.

Table 1 Literature search results

2 Methodology of the research

This section is devoted to presenting the methodology adopted for searching the related work in the area. More specifically, information on search engines, digital libraries, and keywords used in the searching process is provided.

Two search strategies were employed, inspired by the traversing order of breadth-first and depth-first search strategies, respectively. Breadth-first search was performed within a set of predefined published conference proceedings and journals in a specified time period, but only the first level of the search tree was considered for reviewing. Depth-first search was performed by using a search engine to find papers given a set of keywords. A subsequent search was then performed by using the references of those papers as a starting point. This process was repeated until a maximum depth of 3 was reached.

In the following two subsections, more details on each type of performed search are given.

2.1 Breadth-first search

This strategy aimed at finding related papers in a set of recent technical publications, such as journals and conference proceedings, given a specified time frame. The search was performed in the database of conferences and journals of IEEE, ACM, Springer, and Elsevier. Besides, by using the search results, there was also performed a search for all papers published in a given conference or journal by using its table of contents.

The keywords used for the automatic search were (consumer OR personal OR digital) AND (image(s) OR photo(s) or photograph(s) OR photographic archive) AND (value OR quality OR aesthetics OR visual quality) AND (evaluation OR assessment OR analysis OR estimation). A publication period between 2006 and 2012 was defined.

2.2 Depth-first search

In this strategy, the relevant papers were found by using the following methodology: (1) based on a set of keywords, for every result returned by a given search engine, (2) the bibliography was analyzed, and (3) relevant cited work was reviewed, including the root paper itself. This is a practical and useful method for reviewing the literature, once the search is seeded using papers already considered relevant by other researchers. The great advantage is that this method dramatically reduces the searching time for finding relevant papers. On the other hand, there are some drawbacks. First, some stop criteria have to be defined, otherwise this becomes an almost endless process. Second, not every citation is directly related to the research area, since it is common to find papers from correlated areas such as Artificial Intelligence and Neurobiology.

In order to perform this search, some constraints have to be defined. The search is performed in a single level. Another level is considered if and only if a cited paper is strictly related to the area.

2.3 Search results

Table 1 contains the conferences and journals returned from the above mentioned search method. It is also noted if it is a conference or a journal, the publisher name, and the number of papers selected for this survey.

In Table 1, Other refers to conferences or journals with only one related publication. Figure 2 illustrates the balance between conference and journal papers that are reviewed in this survey.

Fig. 2

Number of works published in conferences and journals that are studied in this survey.

2.4 Considerations on the methodology

The methodology previously presented was defined in order to cover relevant papers in the research taxonomy defined in the previous section. Of course, work published prior to 2006, published in low impact conferences/journals or indexed with inadequate keywords might not have been included in this survey. Nonetheless, the number of relevant papers that were included in this survey (150) indicates that a good sample of the relevant work was considered.

3 Enhancement

This section focuses on the research on enhancement techniques applied to digital photography. Usually, the areas of enhancement and analysis work side-by-side, e.g., enhancement is often performed in order to obtain more precise analysis, and a good analysis may help identify which aspects of a digital photo should be enhanced. In spite of that, and for didactic purposes, these areas are discussed separately in this survey.

As mentioned in the previous section, enhancement work may be divided in on the fly and off-line. On the fly approaches are the ones for which it is possible to modify the environment during image acquisition, while in off-line approaches that is not possible, thus, usually the photos are modified or enhanced after acquisition. Nonetheless, both expressions, off-line and on the fly, are also used with other connotations. Chartier and Renaud employed off-line in a noise filtering context [22], while Ercegovac and Lang used the same term in a digital arithmetic context [46].

In the following two sections, more details on photo enhancement approaches are given.

3.1 On the fly enhancement

Although it is generally possible to improve photos by means of a wide range of enhancement algorithms (e.g., red eye correction, histogram processing, among others), there are some particular scenarios from which some information is completely lost during acquisition thus making useless a post-processing operation. Photograph acquisition is naturally a lossy process, which disregards factors such as color, temperature, environment, time and space of the environment, depth of the scene, among several others. For instance, a photo may be considered inadequate due to the zoom choice, e.g., a close-up should not be used when the goal is to show that the subject is in a given location. After the photo is shot, zooming out is not possible, and a good photo might be lost. In some specific situations, it is possible to perform some correction. However, the results are usually far inferior to a scenario in which a new photo could be obtained. For example, image brightness can be adjusted after acquisition in order to improve image aesthetics, but this may result in intensity clipping.

On the fly (or dynamic, live, real-time) enhancement algorithms are proposed to automatically perform or advise adjustments to the camera settings, before the photo is taken. The performed adjustments are intended to improve the photo quality or to avoid undesired conditions, such as inadequate focus and lighting.

Most modern digital cameras have embedded on the fly enhancement mechanisms, such as an exposure meter, an automatic focus adjustment, and a white-balance adjustment. Since those mechanisms are mostly based on low level information, high level information about scene contents is usually input by the user by means of an appropriate scene switch selection. For example, one may adjust the camera scene switch to motion when using camera to shoot a sports scene.

A fully automated system may require that high-level information, such as the location of people in the scene, should be used in order to increase the overall understanding of the scene, as well as to help algorithms to decide where and how to perform changes.

A first example of a fully-automatic approach is the face-priority auto-focus [125], which has been commercially used by several camera manufacturers. The goal is to set the focus of the camera to regions where there are faces, in order to avoid incorrect focus priority. For example, the Nikon D90 camera uses face position to correctly adjust focus on people present in the scene [107].

Another example, which recently became very popular, is the smile shutter function, which shots the photo only when every detected face in the image is smiling. Several cameras and prototypes incorporating the above features have been developed by camera manufacturers, such as Sony and Canon [47].

Photographic composition rules are also considered as important features for the dynamic adjustment of an image. The conformity with photographic composition rules can be achieved with slight movements of the camera. The production of an autonomous robot photographer is a possible direction to take to address this aspect. The robot developed by Byers et al. [13, 14] was designed to be placed in an event, moving towards possible subjects, proceeding the composition, and, finally, obtaining the photo. The subject of a photo may be identified by several approaches, such as considering the output of a skin detection algorithm and a laser range-finder sensor [13, 14]. This information may also be used for finding the path that a robot must follow to reach the subjects. After getting to the desired place, once again the scene is analyzed for achieving a good composition for the photo. Four composition rules (rule of thirds, empty space, no middle, and edge rule) were used to guide their system to obtain a good composition. The system performance was assessed in some real-world events such as a wedding reception and during the SIGGRAPH 2003 conference.

Another example of a robot photographer (but less independent than the one previously presented) is the Sony Party Shot [143]. The Sony Party Shot apparatus can be plugged into the camera for locating and photographing people by moving in three degrees of freedom (pan, tilt, and zoom). The limitation of this approach is the need to put the robot in a fixed position with the aim of locating people of that point of view, and taking photos.

Some approaches may consider acquiring multiple images with different camera settings in order to detect and/or correct issues after image acquisition. For example, two subsequent photos may be obtained with different camera apertures [4]. The acquired images are then combined for finding the subject and analyzing photo composition. Besides improving photographic composition of the image, it is also possible to locate the mergers (which occur due to the projection of a 3D world scene into a 2D representation, in which background objects appear to be connected to the objects in the foreground) [4].

Despite the obvious advantage of on the fly approaches to digital photography, there are a number of limitations. Since most algorithms demand high processing times and some level of scene understanding, on the fly processing might be impractical due to battery consumption. A high processing time can be understood as a period of time that exceeds the time between two scenes setups (e.g., people position and lighting conditions).

In order to improve scene understanding, stereo vision might be employed to find interesting objects, to provide depth estimates, and to improve image segmentation as well, which may help with the acquisition of better photos. In a dynamic environment, stereo vision may be obtained by the use of Pan–Tilt–Zoom (PTZ) cameras through simple-region SSD (sum of square difference) matching [158].

Finally, on the fly composition might be used for automatic and semi-automatic panorama creation [23].

3.2 Off-line enhancement

This section is devoted to discuss existing methods for the enhancement of photos that have already been acquired. The goal is to enhance a photo using only the existing information in the image file (i.e., pixels, Exchangeable Image File format—EXIF information, faces detected, etc.). Changes typically occur at pixel level, and are required when it is not possible to obtain another photo and the resulting photo has room for improvements. For off-line enhancement, the image representation might be in any color space, however Benoit et al. [5] have shown some advantages when using models based on the human visual system for low-level photo enhancement.

Generally, enhancement changes can be used for making more attractive a photo that presents some type of imperfection. For instance, after removing the imperfection that lens dust may cause, a photo may look more attractive [188]. It is also possible to improve photos by smoothing the subject’s skin [77], by adjusting some general aspect, such as contrast and brightness for both generic [183] and specific type of photos (e.g., improving contours in nature photos [128]) or by removing an undesired object (e.g., an unknown person or a light pole [6, 170]). This last type of enhancement may be achieved by image inpainting, as discussed next.

Image inpainting has been largely used for enhancing photos [170]. Through inpainting, one might remove an element which harms the composition of a photo [170]. Inpainting algorithms remove elements from photos by evaluating the surrounding indicated area with statistical analysis and filling this area with a surrounding-like texture. Inpainting may be obtained by combining texture synthesis, geometric partial differential equations (PDEs), and coherence among neighbor pixels [6]. Patch sparsity algorithms are used for improving image inpainting [170].

Enhancement by example has also been employed [72]. When a user classifies a photo, he/she indirectly classifies the features he/she considers important. Processes such as sharpening, super-resolution, inpainting, white-balance, and deblurring are performed in photos so that they reflect the features present in example images. Other types of enhancement are the photo composite, in which a face can be replaced by another [86], and collage algorithms, in which groups selected images in a new one [149].

Cropping algorithms are designed for obtaining an image which has smaller dimensions than the original one. There are several methods for automatic [1, 3, 24, 138, 147, 179, 186] and semi-automatic [133] photo cropping. Cropping is performed by extracting an area of interest from an original photo [82, 147, 179], to improve the quality of the photographic composition [18, 133, 186], to retarget a photo to smaller displays [1, 3, 24, 138, 179], and to recompose the photo [87].

Most cropping methods have in common the use of content-aware strategies. Content detection may vary from face detection [18, 24, 138, 147, 186], saliency detection [1, 3, 24, 87, 147, 186], and user-interaction with tracking of the user’s gaze [133].

Region of interest (ROI) cropping methods intend to remove a fraction of the original image which contains or includes some element of interest. The dimensions of the resulting image are dependent on the original image contents. Some restrictions may apply, such as maintaining the image original proportion or leaving some room from the element of interest to the photo edges [82, 147]. Common applications for such methods are thumbnail cropping and image summarization. A ROI cropping may be improved with face detection for images containing people. However, other elements of interest (e.g., animals) may be detected by using specific detectors [165] or a generic detector such as saliency maps. Saliency and spatial priors have been also used for content-aware thumbnail cropping [82].

Cropping for improving composition may help the photographer to achieve better results by modifying the image dimensions or image proportions. As it is going to be further discussed in Sect. 4.1, there are rules for analyzing the composition quality of a photo, such as the rule of thirds, which may be used in conjunction with small changes in the image dimensions, aiming at better composition by just removing a few pixel rows (or columns) of the photo. On the other hand, there are methods that make direct changes in the image contents. They will be referred to in this paper as recomposition methods, as discussed next.

Cropping algorithms are limited in the sense that they crop the images from the borders. Cropping columns or rows in the middle of the image usually results in distortions. However, in some cases the important content of the image is close to the borders. There is, however, a class of algorithms named retargeting algorithms, which may crop the image to regions other than the borders only.

Retargeting algorithms are mainly designed for adapting an image to different rendering devices, such as mobile phones [1, 3, 24, 138]. The goal is to preserve the main content of the image while discarding unnecessary or redundant information in such a way the main content is more visible than if a simple resample were applied. Global energy optimization for the whole image may be used for image retargeting [127]. Face detection, text detection, and visual attention, all combined, may also be used for finding content in photos [24].

Another class of algorithms, similar to the retargeting algorithms is the class of recomposition algorithms. Recomposition algorithms present a very challenging area of research: the goal is to automatically change the image in order to obtain a more pleasant composition. Some examples of such changes include modifying the subject proportions, removing elements from the photo, cropping the image, etc. Most approaches are, as yet, semi-automatic in the sense they require human intervention to indicate which areas need improvement. In this survey, recomposition algorithms are considered as different from retargeting algorithms since they do not necessarily imply changing the original image dimensions or the original image proportions. The changes are usually artistic ones. Liu et al. [87] proposed a method for recomposition based on finding elements of interest, and applying composition rules (such as rule of thirds, diagonal guidance, and visual balance) to produce a better composed image. In a similar approach, Bhattacharya et al. [7] proposed a semi-automatic recomposition method which uses stress points (adapted from the rule of thirds) for optimal object placement and visual balance for improving composition. Experiments show that 73 % of recomposed images were considered better than original counterparts by human observers.

4 Analysis

In this section, relevant work on photography analysis is presented. Methods in this area may be organized according to their purposes, as follows:

• 1. Assessment. The goal is to score photos on a given scale (e.g., from zero to ten, good or bad) according to some criterion: the image quality (related to some degradation in the image) or the aesthetics of the image could be assessed; and

• 2. Information extraction. The goal is to detect the presence and location of some pre-defined elements of interest, e.g., people and faces, in a photo. The relationship between photos could also be extracted.

In the following sections, each one of these groups of methods is described in more detail.

4.1 Assessment

Assessment algorithms typically assign a score to a photo based on some metric. This allows the creation of an ordering based on the returned metric values. Assessing (or ranking) a photo is a very difficult and controversial task, especially when dealing with consumer ones. Two main aspects can be evaluated: the quality and the aesthetics of the photo. While the image quality analysis, in this survey, is understood as the assessment of the degradation of the image (e.g., sensor noise, resolution, and compression artefacts), the aesthetics analysis is related to the visual appearance and appeal of the photo (e.g., the color harmony and photo composition). IQA is out of the scope of this survey.

Several photo composition techniques and rules of thumb have been defined by experienced photographers, based on heuristics and are considered as responsible for improving the aesthetic quality of a photo. Those rules, known as photographic composition rules, may be used to identify higher-quality photos, based on the assessment of features. Photo composition may be regarded as the most determinant factor to consumers when considering photo quality [134].

The application of a photo composition rule will not necessarily assure best aesthetic results. Notwithstanding, photos obeying such rules are likely to look more appealing to consumers than if they were shot without attention to the rules [17, 134]. However, it is not necessarily true that a photo must have composition rules obeyed to be considered appealing by consumers. This contradiction may be explained by the existence of other factors, apart from composition, e.g., people involved, photogeny and the place the photo was taken.

Some of the photo composition rules were lately explained by the theories of perception. The rule of thirds is a good example: it is known that when the subject of the photo is placed in one of the thirds of the image, the viewer is stimulated, due to the nature of human visual system, to perceive other regions of the photo. Other rules are not well defined in the specialized photography literature or are defined in terms of more subjective concepts, e.g., trying to obtain a more casual and spontaneous picture [12].

Photographic composition rules have been adopted for ranking photos in many researches [4, 14, 17, 18, 39, 41, 73, 81, 87, 95, 139], in which relationships between some predefined rules and the human judgment have been identified. Rules may come from human visual system theories as well as from professional photographer’s expertise.

The rule of thirds is the most explored photographic composition rule in the literature [4, 14, 17, 18, 39, 74, 87]. One of the main reasons for that is because it is easily translated to algorithms. The rule of thirds states that one should preferentially place the subject in a third of the image width or height (depending on the image orientation). Existing works differ on how the subject of the photo is located, for example by using (1) face detection algorithms [14, 17, 18]; (2) low-level information, such as borders and regions found by the mean shift algorithm [4, 87]; or (3) by evaluating the differences of pixels positioned in those interest areas [39].

Other rules are also explored but with less consensus between authors. The zoom rule can be applied to classify photos according to the distance from the camera to the subject. Excessive or insufficient distances are penalized by the algorithm as inadequate compositions. Since a precise detection of the subject is required for this type of analysis, Cavalcanti et al. [18] and Byers et al. [14] used face detection as the main information to identify subject position. In a similar approach, Kahn et al. [74], used the ratio between the area of the face and the area of the image.

The integrity rule was proposed to identify undesired chopping of the main subject. The great drawback of this rule is the high cost of precisely detecting the subject in a photo. The use of anthropometric measures were shown to be effective to subjects in an upright frontal position. Using some reliable information, such as the coordinates, and the dimensions of a detected face [18, 139], it is possible to infer the position of the rest of the subject body, and detect possible chops.

Both zoom and integrity rules were designed considering that there is trusty high-level information such as the face coordinates. This is a great drawback, since an imprecise detection may lead to wrong conclusions. There are approaches that mainly rely on the intensities of pixels rather than on high-level information. One disadvantage is that color images may have their channels treated independently and may result in redundancy which must be treated by classification algorithms. For instance, in the work of Datta et al. [39], 56 initial rules were reduced to just 15 rules after the execution of support vector machines (SVM) and pruning [33], since there was redundancy in applying the same data extraction algorithm in all images.

Finally, the visual balance rule is also used for analyzing if the photo elements are well balanced, i.e., are placed in a photo in a way that observer attention is equally divided by the photo elements [7, 87, 176].

Besides the use of photographic rules there are authors which evaluate low-level features (e.g., sharpness, brightness, and contrast) in order to identify the overall appearance of a photo [39, 73, 80, 81, 108].

Higher level image analysis may also be employed for photo ranking. For instance, aesthetic analysis, may be achieved by learning how humans classify photos according to some subjective criteria. Although that might be difficult, there are studies focusing on the emotions evoked by artwork in humans [174]. The criteria may be diverse. For instance, the time a human spends evaluating an image can be a criterion for confidence on human assessment [45]. It is believed that the emotions evoked by a natural image can be understood by means of aesthetics gap concept. According to Datta et al. [40], “The aesthetics gap is the lack of coincidence between the information that one can extract from low-level visual data and the interpretation of emotions that the visual data may arouse in a particular user in a given scenario.”. Color harmony can also be considered as an important feature to be considered [176]. Low-level information such as lighting, color [74, 80, 95], luminance [74], edges, and range of lightness [48] are used for judging the harmony (a high-level subjective aspect) of a photo and videos [104].

Besides all the above presented factors, there are some other common sense factors that might influence human judgement. Below is a list of these additional factors:

• People involved. A photo may be considered more or less appealing depending on the identity of the shown people, e.g., even a badly composed and illuminated photo might be considered good if it contains people for which the consumer has affection, such as the photographer’s child, a famous person, etc. The opposite might also happen: a well composed photo might be discarded if the person in the photo is unknown;

• Place where the photo was taken. Some photos are related to places rarely visited. Thus, even if a photo has problems, e.g., in composition or illumination, it is likely that it will not be discarded because of its uniqueness;

• Photogeny. Well-composed photos do not necessarily contain photogenic people. It is possible to find one or more group members talking or looking elsewhere in the moment the photo was shot, especially in group photos; and

• Personal preferences. Some people might prefer a photo without obeying composition rules.

Despite the above discussed factors, photo ranking might be useful for helping consumers to identify (at least in a group of pre-selected photos) the ones with more attributes related to a better looking or appealing impression.

4.2 Information extraction

This section includes a discussion on approaches for extracting elements of interest that might be important to a photographic analysis system. The reviewed work involves approaches for face and people detection, landscape analysis (e.g., horizon tilt evaluation), and identification of the image class (e.g., if it is a photo or a graphic image). The goal is neither to rank nor to classify the images but to extract information. This may be considered as an auxiliary source of information for image ranking methods (discussed in the previous section). Elements of interest might be anything the user is searching for: (1) a face; (2) a person; (3) regions with unwanted features such as dissection lines [139], (4) unfocused or blurred regions [151], (5) a sunset area [9], (6) text [153], and many others.

Generally, information extraction by different approaches involves the construction of a classification model for the targeted element (this can be performed, for example, through a learning process using a set of reference patterns). It is commonly accepted that the best technique to build a particular model for a given problem is dependant on specific features of the problem.

Decision trees [43, 130] are typically employed to identify classes that have a reduced number of constraints, both numerical and categorical, such as number of colors, number of people, etc. The ID3 classification algorithm [124] was used for classifying an image as either a digital photo or as artwork (e.g., a logo, a drawing, and other images artificially generated). The decision tree was trained with 1,200 images. An accuracy of 95.6 % was achieved when distinguishing the classes. This result was verified through a tenfold cross validation.

The SVM [33] is largely used for classification, which is useful for detecting features in photographs, such as indoor or outdoor scenes [137], the presence of a sunset [9], the level of expertise of the photographer [152], the presence of skin regions [64], among others. The SVM is normally used when the set of constraints is not small, and there is no clear linear separation of the data for each class. When defining an SVM model, a kernel must be specified. For instance, Serrano et al. [137] used a radial basis function. On the other hand, Boutell et al. [9] and Li et al. [80] used a gaussian function.

When there is a great amount of data, and a great number of components as well, the high correlation between those components may harm classification. Many authors use principal component analysis (PCA) [65] for reducing the dimensionality of the feature space [152].

Some information extraction methods are designed for detecting human-related information, such as face, eye, skin, pose, etc. Since most photos have people, such information is very important to any photographic analysis system.

Recent work in face detection focused on multi-view, rotation, and scale invariant face and eye detectors. Discriminant features [162], low-level features [42], Sobel edge detection, morphological operations, and thresholding [154] may be used for this goal.

Face recognition algorithms can be used for identifying photos which contain or do not contain a specific individual, as well as for finding relationships between images due to the presence of a given person or group [55]. The identification of a specific person might be used, according to a rule defined by Loui et al. [92], to infer the relevance of a given photo based on the relationship of the people to the photo owner. Recognition can be also used, along with human tagging and some logic formalism (e.g., Markov logic), to retrieve social connections in photo repositories [57, 75, 140, 187].

In a photo selection scenario, it is very important to identify the relationship between the people present in some image set. Such a relationship may be used for predicting the significance of such images to that set [91]. This can be done using local patterns of Gabor magnitude and phase [166]. Face recognition highly relies on face detection. Thus, imprecision on face detection may result in a poor face recognition. There are, however, approaches for misalignment-robust face recognition [171].

The use of face details, such as birthmarks [120] and clothes [54], are also used to improve face recognition. A Markov random field is used for recognizing people based on contextual clues such as clothing [2]. Gender can also be a clue for face recognition by means of spatial Gaussian mixture models (SGMM) [84].

Considering that low-level image features are considered by consumers to determine if a given photo is better than another [137], detecting the presence of such low-level features may be very useful for ranking photos. One of those features is the blur. Blur may be used for automatically ranking photos [73, 95]. Blurred images can be identified by the detection of some features, such as image color, gradient, and spectrum information [88]. The spectral analysis of an image gradient is also used for identifying blurring kernels in images [69]. Other features such as clarity, complexity, and color composition are also explored [39, 94].

Besides the face, skin regions are another important evidence for the presence of a human in a photo. Several approaches were proposed. Skin tone may be detected by a pixel-wise approach or by a region-based approach [76]. Both approaches use a color model [64]. Additionally, it might be possible to decompose skin tone in hemoglobin and melanin [169], which can be used for a better understanding of skin texture.

Skin classification may be performed by the use of SVM and region segmentation [64], as indicated earlier in this section. The approaches might be compared with receiver operating characteristic (ROC) analysis [136].

While evidence for people can be obtained by skin detection or face detection, there are also approaches by which humans can be directly detected in images. Recent approaches use local binary patterns (LBP) for human detection through two variants, semantic-LBP and Fourier-LBP [105]. People detection can also be achieved by the use of quantified fuzzy temporal rules for representing knowledge of human spatial data. This kind of data is learned with an evolutionary approach [106]. A head and shoulders detector can be also achieved by the use of a watershed and border detector, whose outputs are used to train a classifier using AdaBoost [168].

Some researches have been conducted for detecting people in a specific context, but might be extended to a more general scenario. For instance, it was shown that the use of region covariance features with radial basis function kernel SVM, and histograms of oriented gradients (HOG) with quadratic kernel SVM outperformed the use of local receptive fields with quadratic kernel SVM in the specific scenario of pedestrian detection [116]. In the same way, the detection of human activities, such as ‘fighting’ or ‘assault’, are recognized and encoded by using context-free grammars through a method which uses a description-based approach [131].

Besides people detection, other types of information may be useful for photography analysis. For example, the social context might be inferred by analyzing the distribution of people found within the image [56]. A graph-based approach has shown to be useful for finding rows of people [56].

The pose of the people is also important information about the photo. Each body part has a limited number of positions when compared relatively to other body parts. For instance, the head is directly connected to the shoulders and might not appear connected to the feet. Thus, if a face is found, the shoulders should come right below. There are several approaches for human pose estimation. Human pose may be estimated in video sequences using multi-dimensional boosting regression from Haar features [8]. In static images, pose can be classified through angular constraints and variations of body joints with the use of SVM [97], with observation a driven Gaussian process latent variable model (ODGPLVM) [61] and non-tree graph models [70], and with a conditional random field (CRF) if multiple views are available. A bottom-up parsing approach can be used to recognize the human body for performing pose estimation by segmenting multiple images.

Besides human subjects, other types of subjects may be considered in a photo. Different subjects (e.g., natural or man-made objects and animals) might appear alone or interacting with humans, resulting in a more complex photo. A shape-driven object detector [59, 129], SIFT [78], and sets of mattes [144] may also be employed for a more general object detector. It is also possible to identify the region-of-interest by using captured camera information stored in EXIF [83].

Instead of detecting a specific type of object, it is also effective to use the identification of regions within the image with some correspondence. In this sense, the image segmentation algorithm has fundamental importance for the photography analysis.

There are several approaches to image segmentation. Since in photography analysis, subjects and scenarios might vary widely, the more general the image segmentation algorithm is, the better the result.

Main methods for image segmentation are based on edge information [19, 159], fragment-based approaches [36, 79], point-wise repetition [182], tree partitioning under a normalized cut criterion [160], a nonparametric Bayesian model [115], a geometric active contour model [180], Markov random fields with region growing [123], Markov random fields and graph cut [25], and local Chan–Vese (LCV) model [163].

Most algorithms deal with both color and gray images. Some image segmentation algorithms are specific to color images [19, 181]. It is normally difficult to compare different image segmentation algorithms, but unsupervised objective assessment methods have been attempted for this task [185].

4.3 Grouping

Photo grouping is designed for setting associations between groups of photos. The associations may be set by either the semantic information found (such as number of faces detected, number of colors, etc.) or high-level information (e.g., Global Positioning System, GPS, position present in some image EXIF).

4.3.1 Classification and clustering

Classification algorithms are designed to identify the class which a given image belongs to. There are several goals for image classification, e.g., (1) identifying, in a set of image files, which ones are photos and which ones are graphics [114]; (2) identifying whether photos were obtained in an indoor or an outdoor environment [137]; and (3) identifying if images were obtained by an amateur photographer or a professional one [80, 94, 111, 151], among others.

It is not completely known how humans perform classification tasks. Vogel et al. [157] have shown, however, that humans use both local and global region-based configurations for scene categorization. This implies that human-inspired algorithms may consider both local and global region-based information for better results in image classification. It was also shown that the color plays an important role in image categorization for humans. Natural images were better classified when presented in color as opposed to gray levels [157]. Classification has a close relation to information extraction, as discussed in Sect. 4.2.

One of the main steps for an accurate image classification is the representation of the image which will later be used as input to a classifier. Representation can be performed by local descriptors [101], a topic histogram using probabilistic latent semantic analysis (pLSA) and expectation–maximization (EM) [93], multilevel representation [164], triangular representation, which is robust to viewpoint changes [67], resolution invariant image representation [161], and scale invariant feature transform (SIFT) [164], among other methods.

Some relevant classifiers proposed in the literature are: AdaBoost [93], SVM [101, 164], multiple kernel learning (MKL) [93], Bayesian belief networks [38], Bayesian active learning [122], and conditional random fields models [15, 184].

The main challenges in image classification are the computational cost and the classification accuracy. A local adaptive active learning (LA-AL) method was used for lowering the number of training samples needed [93]. The within-category confusion can be dealt with probabilistic patch descriptors, which encodes the appearance of an image fragment, and the variability within a category [101].

Clustering algorithms are intended to automatically group images when considering their extracted features. Given a set of photos, clustering can be used to identify the existing relationship between such photos.

Cooper et al. [30] presented an automatic temporal similarity-based method using EXIF data. Graph-based algorithms [52] and local discriminant models and global integration (LDMGI) [173] are common methods for image clustering.

Since clustering is not commonly a supervised process, system improvements are necessary for reducing errors in the system. Thus, user feedback is used as a way to bring out relevant feedback about system performance [11].

4.3.2 Summarization

Another recent area of interest is finding relationships between photos for producing summaries. Summaries are useful since finding information in large sets of images can be time consuming. Summaries are used for producing condensed displays of touristic destinations [117], simplifying photo browsing on personal collections [28, 141, 155], indexing [156], and storytelling [53, 110], among other applications. A specific problem related to the task of producing summaries or filtering out redundant information from a collection of photographs is the detection of near duplicates [28, 126, 148].

Photos matching specific keywords [142] or GPS-tagged information [135] have been grouped to build 3D models of some sightseeing. On-line tools, such as Bing maps [102], used some of those technologies for building 3D models of such places.

4.3.3 Image retrieval

According to Marshall et al. [99], image retrieval techniques can be classed as content-based image retrieval (CBIR) and annotation-based image retrieval (ABIR). In CBIR, the images are processed for obtaining information while in ABIR, images are often annotated with textual information, such as place, time or photographer, and this information is used to retrieve images.

Most detection algorithms can be used as an intermediate step for retrieving images in CBIR [90, 99], such as recognized faces [121, 187] and events [37], among others.

Since manually tagging photos can be time consuming, recent work considers the use of information automatically obtained from EXIF [83, 85, 132, 148], SIFT [26, 148], face recognition and connections found in social networks [27, 113, 155], and georeferences, which might be obtained from GPS devices [16], people clues such as faces and clothes [146] or other high-level information [132].

4.4 Discussion

In this section, algorithms for photo analysis have been organized in three categories: assessment, information extraction, and grouping. From the performed review, assessment seems to be the less explored area. This may be explained by the highly subjective nature of the task, which makes it difficult to perform precise or universal analyses. The other two areas are more explored in the literature and present a richer set of approaches.

Besides the underlying limitations discussed in the next section, the approaches seem very promising to be included in a photography analysis system.

5 Critical analysis

The main issues covered in the studies reviewed in this survey are considered in this section. To better discuss such issues, the following information about the articles were summarized: the source of the used image set, the size of the image set, the main goal, the metrics used for assessment, and the achieved results. The photo analysis algorithms are shown in Table 2 and the enhancement algorithms are in Table 3.

Table 2 Summary of the reviewed work on analysis techniques

Table 3 Summary of the reviewed work on enhancement techniques.

This section contains two subsections. In the first one, a review of the image sets used in the experiments is given. In the second one, commentaries about the validation processes are presented.

5.1 Image sets

For most of the image analysis algorithms reviewed in this survey, the purpose is to perform tasks in a human-like manner. Thus, it is fundamentally important to ensure the photo sample is representative for testing.

Some studies were performed to identify the user behaviour when photographing [96], sharing [103], analyzing [49], and managing [35]. However, based on the conducted literature review, strong evidence about the user preferences were not drawn, the assessment of most algorithms for photo enhancement and analysis are performed by means of subjective assessment.

According to the conducted literature review, there is no defined methodology for carrying out subjective experiments for photos analysis. Some methodology ought to be employed due to the number of factors that might influence subjective assessment. Some of these factors are:

• 1. People involved. While in professional photos, the people present in the photo are usually part of the subject, in consumer photos, people are mostly known and significant to the photo owner. Therefore, a photo assessment performed by consumers might be too strict in the absence of a known person and too flexible in the occurrence of, for instance, a family member;

• 2. Place and event. In some situations, the photo might not be technically good, but captured a place or a rare event. This could positively influence the judgement of the photos;

• 3. Style used. Different users adopt different photo habits. The individual style of a user might not be appreciated by other users;

• 4. Number of Images. There are an endless number of poses, camera settings, and subject positioning. Therefore, it is barely infeasible to represent this diversity of possibilities in a small set of images.

In Tables 2 and 3, the second column (Image sources) indicates the databases from which the images were obtained. In this column, Web refers to web crawled images and Own refers to particular photos from the authors or contributors. The third column, (Set size), represents the number of images used in the experiments (if any). The fourth column indicates the main goal of the work. The fifth column briefly indicates how the approach has been evaluated, in which Obj. represents an objective assessment and Sub. a subjective assessment method. In the final column is shown the best reported performance of proposed algorithms. Tables 2 and 3 have been built based strictly on what was described in the papers. Whenever the information was not explicitly shown in the paper, results are shown in a non-numeric way or the information is not suitable for the discussed problem, NI (Not Informed) is used. Both tables are sorted based on the total number of images and then alphabetically by the name of authors.

By analyzing the Tables 2 and 3 it is possible to draw some conclusions about the number of images and their sources. First, there is no consensus on the database to be used. This makes it impossible to perform a direct comparison between the results in Tables 2 and 3, and to reproduce the experiments as well. Second, the number of images employed in the evaluations drastically vary. The average number of images employed in photo analysis work is 45,344 with a standard deviation of 212,556, and a median of 3,581 with an interquartile range of 12,278. If only photo enhancement work is considered, the average is 716 with a standard deviation of 952, and a median of 375 with an interquartile range of 766. Third, no work presented a categorization of the image set, e.g., not known is the distribution of the number of people among a given set. Finally, some papers only presented a simple visual verification of the results (e.g., Achanta et al. [1] and Banerjee et al. [4]).

It is important to highlight the non-utilization of a labeled and representative public image database for photographic analysis. Therefore, most authors crawled images from on-line repositories. Web crawlers can be employed for creating image sets which present a richer and diverse number of situations, and a higher number of pixels [24, 114, 137, 139]. Nevertheless, the great drawback is the lack of copyright licenses for public experiments. There are some public image databases that are free for academic research use (such as Flickr [50] and other databases under Creative Commons license [34]), yet they are not labeled. Regarding photo analysis, there are some web databases which have been used as a ground-truth for subjective quality analysis (e.g., DPChallenge [20] and Photo.net [60]). However, since those databases were designed for photo contests, they typically do not represent the reality for consumer photography, which usually have less quality and less exigent evaluators.

Two authors have built datasets in order to make them available to the community. The first work, from Luo et al. [94], presented a dataset of 17,000 labeled photos. The set was built to be diverse, once photos are distributed over seven categories, they were labeled as high or low quality. The problem of this photo set lies in the labeling process. Some important information is not shown, such as the exact number of votes for each category, the origin and background of the photographer, and the personal information of the voters. Besides this, a more precise ranking (instead of only classifying images as high/low quality) could be used for a more general use in enhancement and analysis algorithms. The other work, from Bhattacharya et al. [7], presented a smaller photo set (only 632 images). Other factors, such as the ones analyzed on the Luo et al. [94] approach, could not be evaluated, since the image set built by Bhattacharya et al. [7] could not be downloaded due to a Web server error. Thus, it might be considered that the image set is no longer available.

One might suggest that if it is possible to learn an expert opinion about a photo, it would be possible to analyze a photo. However this is surprisingly not always true. Since average photography consumers do not have training in what a good photo is, they often do not agree with advice given by experts. There are several other factors that might influence a photography user’s opinion, such as photo effects and the event from which the photo was obtained, rather than photographic rules.

In conclusion, it was not possible to identify comparative studies involving different approaches, which considered publicly available photo datasets. This causes difficulties to reliably compare techniques when dealing with consumer photography. The use of image sets from photography contests has also its disadvantages since both photographers and voters may have professional skills or are highly interested in photography. This may lead to results that are not related to ordinary photography consumer preferences.

5.2 Validation

This section contains a discussion on validation approaches. Photography might be considered an art form [66]. There is no simple way of deciding whether a photo is aesthetically pleasant or not. However, it might be possible to identify some metrics that would help photo assessment, and that would be a step further in this area.

Another important aspect to be considered is how approaches were validated. Since the reviewed work is about photo enhancement and analysis, the results are usually images (in the case of photo enhancement algorithms) or abstract information, e.g., color/gray-scale maps, statistics, and scores. Both have a very high subjective component, although some metrics might be defined for obtaining a more objective analysis in a specific scenario.

Validation methods can be classified as subjective or objective. Subjective methods involve subjective experiments in which humans are asked to give their opinion on photos of a pre-defined test set with respect to a given attribute or criterion. A participant may give his/her opinion based on the following methods [98]:

• Single-stimulus rating. The participant will give a score to a photo or a group of photos. The score might be continuous (such as 0–10) or categorical (e.g., excellent, good, fair, bad, and poor). During the rating process, each photo is typically showed to the participant for a fixed presentation time (e.g., 3 s);

• Double-stimulus rating. While analogous to the single-stimulus rating, in double-stimulus trials a reference photo and a test photo are presented in random order, one after another, for a fixed presentation time (e.g., 3 s);

• Forced-Choice. The participant is forced to choose only one within a group of photos, according to a given criterion;

• Pairwise similarity judgement. Similar to forced-choice but, besides choosing one from a group of photos, the participant has also to indicate on a continuous scale how large the difference in quality is between the two photos; and

• Indirect. The participant does not directly give his/her opinion. The quality may be inferred by some measurement such as the time needed for the participant to choose a photo.

Other details and comparing methods can be found in the work of Mantiuk et al. [98], in which a comparison between the first four above-mentioned methods is given. The better method is usually the one with higher correlation between human and automatic labeling. It was shown, however, that for comparing IQA algorithms, in which differences between images might be small, the forced-choice pairwise comparison is the most accurate and time-efficient [98].

Besides the comparing method, there are also other factors have an influence in the experimental assessment, since such experiments involve humans. Some of the factors are:

• Number of participants. Once the opinion about the quality of a photo may vary from one person to another, it is important to have a large number of participants in order to identify features that are more significant in human analysis;

• Used equipment. When the experiment is conducted in an uncontrolled environment, the equipment used might harm the results (e.g., the calibration of the screen in a color experiment might produce a different opinion);

• Knowledge in photography. Experts evaluate photos in a different way than consumers do. As an example, professional photos might be considered good by both expert and consumer while a consumer photo might be considered good by a consumer but bad by experts;

• Cultural diversity. The style and subject of the photo might influence the judgement depending on the participant’s background and origin; and

• Number of photos. The number of photos in the experiment is a factor as crucial as the number of participants. If, on the one hand, a great number of photos might better represent the diversity of the photos, on the other hand, it might reduce the number of volunteer participants, since it becomes a more laborious experiment.

Since, according to the literature review, there is no database which considers all those factors, most of the conclusions drawn from subjective experiments might be considered partially biased. Besides the drastic influence of such factors, there is no consensus on what are the ideal values for them. Thus, most papers present some questionable decisions on the validation step, such as the number of participants (e.g., three participants [108]), knowledge in photography (e.g., most participants are experts [73]), and the number of images used (e.g., only 34 photos to represent the analysis sample [49]).

On the other hand, objective metrics present a set of well-defined criteria, and proposals are evaluated based on those criteria. For instance, the best algorithm might be the one which has lower false-positive rates in a face detection scenario.

Objective methods are usually less expensive since they do not rely on the availability and classification coherence of human participants. However, there are some important features that are not yet well-assessed by computational algorithms, such as the global visual aspect of a photo. Even humans may disagree with a classification result, what may imply a harder subjective assessment. Both approaches (objective and subjective) are important, each in its specific application scenario.

As it can be seen in Tables 2 and 3, the methodology of the assessment widely differs in the reviewed work, with regards to the following aspects: (1) the assessment method, (2) the metric used for assessment, and (3) the source of the photo set.

Although the results reported in those tables were obtained with different algorithms and different goals, it is possible to conclude that most approaches have opted for subjective assessment when dealing with image enhancement and analysis. The reason is probably the lack of consensus on the image set to be used as ground-truth and the essentially subjective task of comparing images.

There is also a lack of clarity regarding the number of people used in the subjective experiments, their confidence with the labeling, and the methodology of the experiment.

6 Conclusions

This survey reviewed state-of-the-art methods for photo enhancement and analysis. For better understanding of this research area, a taxonomy was defined based on the related work. The main conclusions of this survey are discussed next:

• According to the conducted literature review, this is the first survey on consumer photographic enhancement and analysis techniques;

• The interest in algorithms for photo enhancement and analysis has been growing recently, based on the number of recent papers published in this area;

• There is not a consensus on a methodology for conducting subjective photo analysis experiments;

• Although the results were obtained with different algorithms and different goals, it is possible to conclude that most approaches have opted for a subjective assessment due to the lack of a public and labeled image set that might work as a ground-truth for an objective assessment, and due to the inherently subjective task of comparing images. Therefore, in this scenario, direct comparisons between existing approaches might be unfair;

• Some work that indicates the photo sources are not reproducible, since the photos used for testing are not clearly identified due mostly to copyright reasons or the great number of images;

• There is no consensus on the number of images to be used in the experiments; and

• There is a lack of clarity regarding the number of people used in the subjective experiments, their confidence with the labeling provided, and the assessment methodology.

Thus, it is possible to conclude that, although there has been recent growth in photo enhancement and analysis techniques, this is an area with large potential. Experimental assessment needs to be improved, and assessment methodologies are required as well in order to obtain strong conclusions about methods and results.