If you look at all the posts on the various forums, you will see, again and again, that the biggest health-issues reported with a RVAF diet are to do with raw dairy. And that's just those who are honest about it. I have encountered another large proportion of people who claim that raw dairy is the ultimate "detoxing" food, and one find out, again and again, that such people only experience such nasty side-effects when they are consuming raw dairy as opposed to other raw foods....
I agree with Van and Tyler and I've noticed the same things Tyler has among raw dairy eaters and also among raw vegans--especially extreme raw fruitarians. You only have to compare the forum posts of raw vegan/fruitarian and AV followers to the posts of RPDers here to see that we RPDers tend to have far less "detox" issues like constipation/diarrhea, anxiety, depression, dry skin, acne, etc., etc. Like Tyler indicated, most of the "toxins" that need to be detoxed come from the foods the people are eating.
Most of the people I've encountered who are still having problems with Paleo are doing one or both of two things:
1) eating dairy (raw or otherwise)
2) eating lots of heavily-cooked foods
Then there are others who have pre-existing glandular problems with thyroid or adrenals.
As for the raw vegans, one guy (Amir) who tried a raw vegan diet found he was getting negative symptoms from it (he was healthy before he started) and posted about his concerns at a raw vegan forum. The other forum members essentially castigated him for worrying and complaining about nothing and told him it was detox. He asked them why the people in this forum didn't seem to have nearly as many of the nasty "detox" symptoms, and instead seemed to report all sorts of benefits. They ignored his points and just dismissed them with scorn and laughter.
Who knows, some folks may be able to handle raw dairy without problems, but given that it's not part of our evolutionary heritage I think the caution principle suggests we should skeptical of it until it's proven healthy, unless it's simply impossible to obtain any healthier fats. If I were hell-bound to try raw dairy I would get rid of all negative health symptoms with RPD first, then I would try a raw dairy experiment, and even then it would only include raw dairy fat, not raw dairy milk. Raw dairy is too expensive and inconvenient for me now to bother with it anyway. However, my market's grassfed suet price shot up from 90c/lb to over $5/lb again at the last visit. If that price holds I may need to reconsider.