I agree that each person should find the diet that works best for them. I however, believe that those who are vehemently against dairy should also take into account that, due to poor nutrition of our recent ancestors, many of us do not have the microflora or the healthy gut lining to digest milk protein or lactose -- yet this doesn't mean that one's gut cannot recover.
Dr. Natasha Campbell McBride, Neurologist and Nutritionist, has been treating children with autism with a Weston Price diet called the GAPS Diet or the Gut and Psychology Syndrome Diet and has had great success. She makes sure not give raw milk first, but instead recommends fermented raw dairy like kefir and yoghurt. This insures that the body is provided with the bacteria needed to digest lactose. In addition, her treatment is all about healing the gut, which in her belief is the cause of many psychiatric, behavioral, and learning disorders.
For children and adults without proper intestinal microflora digesting their food and permeable intestinal linings allowing proteins to escape into the blood, modern foods can be hazardous. Hence the onslaught of deadly allergies, gluten intolerance, and in my opinion, milk sensitivity.
Mother's breast milk plays an important role of transferring probiotic bacteria to a child, but if the mother does not have this bacteria, she does not have anything to give. What you are left with is babies (and later teenagers and adults -- myself included) with digestive disorders due to a sterile intestinal ecology. And then all too often, doctors give these children antibiotics, further worsening the problem. And of course, the same thing can occur to children who are not breast-fed and not supplemented with probiotic bacteria.
Another thought I have about dairy is the Gerson Cancer Therapy and the Raw Milk Cure. A quick google search for either of these will provide a wealth of information for treating chronic disease with raw dairy. Raw Milk Kefir in the case of Gerson's cancer therapy and simple Raw Milk for Tuberculosis and other diseases was found to be effective in many cases. In fact, to my understanding the Mayo Clinic was founded to give the Raw Milk Cure.
All in all, if dairy doesn't work for you, then definitely avoid it, but that doesn't mean that some people don't thrive on it. The two tribes in Weston Price's research seemed to, as well those tribes of the Caucasus Mountains who drank large quantities of Milk Kefir daily.