On Sárközy's theorem for shifted primes

Suppose that $A\subset \{1,\dots ,N\}$ has no two elements differing by $p-1$, $p$ prime. Then $\mathrm{\mid }A\mathrm{\mid }\ll N^{1-c}$.