Well, the good news is, dairy isn't a very bodybuilding friendly food group. I don't eat much dairy, nor do most people I know. Milk has a lot of sugar in it and overall, a high dairy diet leaves an inflammation or film over the body. Some people are effected worse than othes. There are a lot of sugars in various dairy products, but I also believe that this response has to do with various levels of lactose intolerance. I believe that just because someone isn't having noticable stomach upset doesn't mean that they aren't experiencing a negative response to dairy. It sounds like you are very lactose intolerant and I am sure that you are better off without it. This would probably mean that when you use whey protein, you will want to stick with only high quality whey isolate, as they are lactose free. Outside of that, there is no real reason to use dairy, especially when dieting down