Vitamin D is more important as Vitamin D deficiency can cause osteoporosis. Vitamin D regulates calcium levels.
Also, many plants and some animal foods (bones, small fish) are good sources of calcium. I think over-domestication and soil depletion is why there isn't as much calcium, for example the wild fruit baobab has more calcium than milk.