Hi Lee
Dont worry too much! While it's true they have recorded some of the strongest winds in the country there (close to 200mph I think) on a normal day it is not much of an issue (in fact Ventoux doesnt come from 'vent' contrary to popular belief) , at least the four times I have done it. Even on the way down I have not had to be unusually careful about being blown off the road.
Of course if it's windy in the valley, it's likely to be very windy up there (yes i have seen that video of guys hanging onto their bikes!)
Late July it's likely to be pretty hot (done it once last year by 40c at the bottom, it was still 30c at the top, it felt too warm all the way up, its only when you stand at the top in sweaty kit that you feel chilly). If it's about that warm you could get away with just armwarmers and some newspaper to stuff under your jersey for the way down and standing at the top (dont plan on staying ages though), otherwise to be safe a windbreaker.
Have fun - if you are there for some time it's worth doing the other side from Malaucene too- less well known but I found just as hard, and with different views. The climb from Sault is much gentler for most of the way- one to keep for the third day maybe