ControlFill: Spatially Adjustable Image Inpainting from Prompt Learning

In this report, I present an inpainting framework named $\mathit{\text{ControlFill}}$, which involves training two distinct prompts: one for generating plausible objects within a designated mask ($\mathit{\text{creation}}$) and another for filling the region by extending the background ($\mathit{\text{removal}}$). During the inference stage, these learned embeddings guide a diffusion network that operates without requiring heavy text encoders. By adjusting the relative significance of the two prompts and employing classifier-free guidance, users can control the intensity of removal or creation. Furthermore, I introduce a method to spatially vary the intensity of guidance by assigning different scales to individual pixels.