I'd agree that they would not be tough to break down like something with a lot off fiber, maybe something like raw yam, but on a cellular level your pancreas is going to have to expend the energy for the digestive enzymes, and of course any time you cook something, toxins are going to be generated so that's going to add a load to your immune system directly. Plus the chances are better than good that the protein is coming from a less than ideal source (cafo, farmed fish, etc) so that's some more bad juju there.
Just some thoughts, do what works for ya but maybe your body is saying it doesn't like that and you're not reading the signs.