Thats it!
Though conditions varied greatly.... One has to suspect that many Paleo people with large robust frames did not work from dawn to dusk as many agrarians had to. Fisherman tribes of the physically powerful Cro-magnon, could exert great energy and strength in small burst, such as pulling in a hall of fish, then spend the rest of the day doing light work, such as tending and mending the basic tools and amenities. Since they had the raw materials to build powerful bodies, and greater energy, through better nutrition, they would not necessarily have to work so much more strenuously than the agrarians in order to maintain a much more robust skeletal frame. Their hormonal systems being perfectly balanced, they could build up strong muscles and bones without endless hours of pumping iron. It may be possible that because many of them were overflowing with health and life energy, that they would, like other primates burn off extra energy and build up great strength by simply playing around, or wandering about- not out of some necessity, but rather as a pleasant way to spent their leisure time and excess energy.
On the contrary, the agrarian persuasion, no matter how much physical toil they underwent, without the proper nutrition provided by our evolutionary diet, they would inevitably be stunted. There is plenty of evidence that suggest that their weakened bone structures could be a result of poor nourishment, and not lack of exercise. Many feudal surfs worked themselves into decrepitude in order to keep up with the physical demands of agrarian society. They were unable to regenerate fast enough to keep up with the physical demands, and many skeletons are found of young neolithic people with rotten teeth, and degenerative bone disease.
There are signs of bone degeneration, along with the inability to regenerate, in neolithic man, which point to evidence that there must be some nutritional / other environmental components responsible for the weakening of our bone structures.