Ask, Attend, Attack: A Effective Decision-Based Black-Box Targeted Attack for Image-to-Text Models

While image-to-text models have demonstrated significant advancements in various vision-language tasks, they remain susceptible to adversarial attacks. Existing white-box attacks on image-to-text models require access to the architecture, gradients, and parameters of the target model, resulting in low practicality. Although the recently proposed gray-box attacks have improved practicality, they suffer from semantic loss during the training process, which limits their targeted attack performance. To advance adversarial attacks of image-to-text models, this paper focuses on a challenging scenario: decision-based black-box targeted attacks where the attackers only have access to the final output text and aim to perform targeted attacks. Specifically, we formulate the decision-based black-box targeted attack as a large-scale optimization problem. To efficiently solve the optimization problem, a three-stage process $\mathit{\text{Ask,}}\mathit{\text{Attend,}}\mathit{\text{Attack}}$, called $\mathit{\text{AAA}}$, is proposed to coordinate with the solver. $\mathit{\text{Ask}}$ guides attackers to create target texts that satisfy the specific semantics. $\mathit{\text{Attend}}$ identifies the crucial regions of the image for attacking, thus reducing the search space for the subsequent $\mathit{\text{Attack}}$. $\mathit{\text{Attack}}$ uses an evolutionary algorithm to attack the crucial regions, where the attacks are semantically related to the target texts of $\mathit{\text{Ask}}$, thus achieving targeted attacks without semantic loss. Experimental results on transformer-based and CNN+RNN-based image-to-text models confirmed the effectiveness of our proposed $\mathit{\text{AAA}}$.