The number of composite class sizes in finite groups

We consider finite groups with at least three conjugacy class sizes that are composite numbers and we prove that, in that situation, the number of prime class sizes is bounded by the number of composite class sizes. The analogous result for (non-)prime-power class sizes is also addressed.