Evolutionary Black-box Patch Attacks on Face Verification

University of Information Technology - VNUHCM
✅ GECCO 2025 (Accepted Poster)
Overview

A small adversarial patch placed near the nose significantly lowers the similarity score between two images of the same person (similarity = 0.4938), causing the model to misclassify them as Different Person. In contrast, the clean image and another pose variant—without any patch—are correctly identified as Same Person (similarity = 0.8933). This highlights how localized patch attacks can effectively fool the model with minimal visual changes.

Abstract

Deep learning-based face verification systems are typically vulnerable to adversarial attacks, i.e., unnoticeable perturbations of input images making neural network models misrecognize images of the same person. Generating adversarial attacks to evaluate the robustness of these systems is crucial for their reliable deployment. However, most effective attack methods focus on classification tasks and operate in the white-box setting, where attackers can access the gradient information and the internal architecture of victim models. Such unrealistic scenarios overestimate the adversarial risk. We investigate the potential of crafting adversarial perturbations in the black-box setting, where attackers can observe only input images and output responses of victim models. We employ the genetic algorithm (GA) to search for adversarial patches that can camouflage well into facial images. The search involves two conflicting objectives: attack performance and reconstruction quality. We consider four GA variants: a GA optimizing the combined fitness function of the two objectives, a GA that favors well-blended adversarial patches, a GA that focuses on attack performance, and a multi-objective GA that optimizes both objectives separately and simultaneously. Our methods demonstrate strong performance in attacking face verification systems under the realistic black-box setting and generate more natural-looking patches compared to baseline approaches.

Method Overview

Pipeline

Figure 1: Creating and evaluation of an adversarial patch using Genetic Algorithm (GA)

Architecture

Figure 2: An example Pareto front of NSGA-II. Green points represent successful attacks, while red points represent unsuccessful attacks. The highlighted point in the middle represents the successful attack with the highest \(\mathcal{F}_\text{recons}\).

Experimental Results

Result A

Table 1: Patch size \(20 \times 20\), population \(N = 80\), mutation \(0.5\).

Result B

Figure 3: Adversarial and PSNR scores of the proposed approaches across \(10,000\) iterations (i.e., generations).

Experiment Result 2

Figure 4: Comparison of result images generated by different algorithms. Green borders indicate successful attacks, and red borders denote failed attacks.

BibTeX

@inproceedings{zz,
  author       = {Khoa Tran, Linh Ly and Ngoc Hoang Luong},
  title        = {{Evolutionary Black-box Patch Attacks on Face Verification}},
  booktitle    = {GECCO '25 Companion: Proceedings of the Genetic and Evolutionary Computation Conference Companion},
  address      = {Málaga, Spain},
  publisher    = {{ACM}},
  year         = {2025}
}