I can see if someone is eating a conventional diet and happy taking supplements and so forth, it might be somewhat wise to avoid the heavily loaded wild fish and of course everyone should avoid and not support farmed fish in general.
I know some people will say that fish and sea foods aren't really a necessary part of our natural diet, but not only is that very speculative, in today's reality fish and sea foods are often one of the major available sources of wild food. Cutting them out and being without game meats would mean a diet mostly of domesticated animals which would - I assume - garner some environmental and degenerative toxicity in comparrison to wild foods hundreds or thousands of years ago.
If people want to stick to the smaller end of the chain, that is their business, but I don't think it makes much sense. I tend to agree that alot of it is fear-mongering and the logic is somewhat akin to the ideology of eating a plant based diet because this is what the grazing animals eat and that the meat is then 'toxic' and indirect. Unless one believes we are meant to eat the algaes and plankton, odds are the bivalves and fatty fish are more our style. Unfortunately the stream salmon and so forth that probably represented a more natural food source than the deep ocean fish ARE probably no longer very edible due to pollution.
I believe the the mercury and pollution issue is very real as well in terms of ocean fish, at the same time, eating a raw food diet high in minerals and fats -including sea foods - should go a long way towards removing these sort of toxins from the body, to the point that minimizing risk seems more a preference of extreme fussiness than a necessity. Unless I was very careful about the exact PPM of my directly sourced spring water, I wouldn't think to drop my intake of swordfish and tuna, but I honestly would probably not base my diet around those things either.