Difference sets and the primes

Suppose that A is a subset of {1,…,N} such that the difference between any two elements of A is never one less than a prime. We show that |A| = O(N exp(-c(log N)^{1/4})) for some absolute c>0.