Efficiently Detecting Plausible Locations for Object Placement Using Masked Convolutions

Abstract

Being able to insert new objects into images is an important problem for both artistic image editing and for data augmentation. For a successful image manipulation, the plausible placement and the blending of the new objects in the image are critical. In this paper, we propose a fast method for the automatic selection of plausible locations for object insertion into images. Like previous work, we approach the object placement problem as a detection problem – given a bounding box, we evaluate whether an object is present inside the box based only on the neighborhood of the box. However, previous work requires a forward pass for each potential bounding box location. We propose instead to make use of masked convolutions to compute featuremaps for left, right, top and bottom contexts just once per image. Combining these features in such a way that no information from inside a bounding box is propagated to the final classifier allows the model to evaluate a grid of proposals on the featuremaps rather than on the image, speeding up inference dramatically. We validate that our model can generate plausible placements using experiments on the COCO dataset and on a user study. Our method trades off speed for performance, as compared to a patch based approach.