Benchmarking and Analyzing Generative Data for Visual Recognition

Advancements in large pre-trained generative models have expanded their potential as effective data generators in visual recognition. This work delves into the impact of generative images, primarily comparing paradigms that harness external data (\ie generative \vs retrieval \vs original). Our key contributions are: $\mathbf{\text{1)}}\mathbf{\text{GenBench}}\mathbf{\text{Construction:}}$ We devise $\mathbf{\text{GenBench}}$, a broad benchmark comprising 22 datasets with 2548 categories, to appraise generative data across various visual recognition tasks. $\mathbf{\text{2)}}\mathbf{\text{CLER}}\mathbf{\text{Score:}}$ To address the insufficient correlation of existing metrics (\eg, FID, CLIP score) with downstream recognition performance, we propose $\mathbf{\text{CLER}}$, a training-free metric indicating generative data’s efficiency for recognition tasks prior to training. $\mathbf{\text{3)}}\mathbf{\text{New}}\mathbf{\text{Baselines:}}$ Comparisons of generative data with retrieved data from the same external pool help to elucidate the unique traits of generative data. $\mathbf{\text{4)}}\mathbf{\text{External}}\mathbf{\text{Knowledge}}\mathbf{\text{Injection:}}$ By fine-tuning special token embeddings for each category via Textual Inversion, performance improves across 17 datasets, except when dealing with low-resolution reference images. Our exhaustive benchmark and analysis spotlight generative data’s promise in visual recognition, while identifying key challenges for future investigation.