User Preferences Modeling and Learning for Pleasing Photo Collage Generation
User Preferences Modeling and Learning for Pleasing Photo Collage Generation
We consider the problem of how to automatically create pleasing photo collages: given a set of photos and a canvas area, we want to arrange the photos on the canvas in a pleasant unsupervised manner. Assuming that the size of the canvas area is smaller than the sum of the sizes of the photos to be displayed, two main issues arise. The first is that photos may occlude themselves, the second is that photos may be partially outside the canvas area. To address these issues, taking into account the pleasantness of the resulting collage, we must consider the order with which the images are placed on the canvas, and their spatial arrangement. Usually, the most important photos are placed at the top of the less important ones in order to minimize the risk of being severely occluded, and composition properties related to photo contents, geometric constraints and aesthetic consideration are taken into account to maximize the pleasantness of the resulting collage. The criteria that define what is important in a photo and what composition properties should be satisfied may vary from user to user. Moreover the single criteria may compete against each other. To be used in an automatic system for photo collage generation, the pleasantness criteria and their relative importance must be properly quantified using suitable algorithms. A the end of this process a fitness function can be defined whose value represents the overall degree of pleasantness of a photo collage. To obtain the most pleasant collage, an automatic algorithm must search the best arrangement of the images by maximizing the value of the fitness function. To this end an optimization algorithm is usually exploited. Several formulations of some of the above criteria have been proposed in the literature but none of the exiting works performed an user study in order to actually determine what are the criteria that made a photo important, what constraints must be satisfied in order to have a collage balanced, or what hints users pay attention to in judging the pleasantness of a photo collage. We argue that if we could elicit the criteria by modeling the users’ preferences, we would be able to create more pleasant photo collages.
Example Images
The first three images are collages created using a single informativeness map and the basic criteria. The fourth image shows the collage created with the learned informativeness map and criteria.
BURST. Saliency map. Basic criteria.
BURST. Quality map. Basic criteria.
BURST. Harmony map. Basic criteria.
BURST. Learned map and criteria.
FASHION. Saliency map. Basic criteria.
FASHION. Quality map. Basic criteria.
FASHION. Harmony map. Basic criteria.
FASHION. Learned map and criteria.
LAND. Saliency map. Basic criteria.
LAND. Qualiry map. Basic criteria.
LAND. Harmony map. Basic criteria.
LAND. Learned map and criteria.
SELF. Saliency map. Basic criteria.
SELF. Quality map. Basic criteria.
SELF. Harmony map. Basic criteria.
SELF. Learned map and criteria.
ZEN. Saliency map. Basic criteria.
ZEN. Quality map. Basic criteria.
ZEN. Harmony map. Basic criteria.
ZEN. Learned map and criteria.
Generalization of the learned criteria
These collages have been created using the already learned criteria of the previous images. Top: collages created with a single informativeness map and basic criteria. Bottom: collages craeted with the learned informativeness map and criteria.
PIXELS. Saliency map. Basic criteria.
PIXELS. Learned map and criteria.
PRIMARY. Saliency map. Basic criteria.
PRIMARY. Learned map and criteria.
RED. Saliency map. Basic criteria.
RED. Learned map and criteria.
SELFIE. Quality map. Basic criteria.
SELFIE. Learned map and criteria.
SHALLOW. Saliency map. Basic criteria.
SHALLOW. Learned map and criteria.
SILHOUETTES. Saliency map. Basic criteria.
SILHOUETTES. Learned map and criteria.
Publications
1.
User Preferences Modeling and Learning for Pleasing Photo Collage Generation (Simone Bianco, Gianluigi Ciocca)
In Transactions on Multimedia Computing Communications and Applications, volume 12, number 1, pp. 1-23, ACM, 2015.
