Your childhood sounds quite wonderful. My parents are very cultured but somehow it didn't rub off on us children (although we did go to Oxford, so something stuck). It used to amaze me that my mother and father could ski, ride, shoot, play the piano, the violin, that when they watched 'Name that Tune' they could always name the obscure classical piece from watching hands on a dummy wooden keyboard (a few rattling finger movements, a few thuds, and "ah yes, isn't that one of Chopin's piano interpretations of Voríšek's impromptu 7th Opus ... etc.", that when they watched Mastermind they would often beat the contestant on his or her specialist subject, yet we children could do none of these things. (And still can't - although we are a wicked pub quiz team.)
(Truthfully as a child I always wanted grow up to be like Caractacus Potts in Chitty Chitty Bang Bang (and be an astronaut). Your dad sounds a bit like that. :-D )
That sounds a lot like me. My grandparents and father did such marvelous things, but I, myself, seem to haven't. I'm at UMBC, which is probably the least cultured (student-wise) college anyone in my family has gone to. (My paternal side was all Yale and Princeton. Maybe one or two at Harvard.) I've always been awful at music--although I listened to classical and orchestral music as a child. Skiing is still my favorite sport. A good friend of mine is a horsewoman, so I really should ride more than I do. I've taken lessons in many things--riding, piano, tennis, art, but it never entirely stuck. Although I suppose language did. I'm always constantly learning a new one. (Tolkien is one of my favorite authors for his philology; he certainly influenced me as a child. Studying languages is a hobby of mine.)
I was raised, primarily, by my grandmother, who was an artist, and my father, a geologist. So my grandmother was pretty much in charge of culture and the arts, while my father was history and science. I'm an OK artist, but I believe that's primarily because I don't do it often enough. My father tried to teach me how to shoot, but at the time, I was much too small (in size, not age--at roughly 100lbs, its hard to handle a rifle), but I still want to learn. I know he hated hunting, for a variety of reasons, but primarily because its boring. If you're a Class B skeet shooter, then its hard to enjoy dove hunting when you hit your target all the time.
It's always great to see someone else who had a highly cultured family. No matter how much it rubs off, it seems as though you keep that respect for it. A few of my friends don't know much about art, for instance, and have always loathed art museums, but then I might show them a painting or two that I love, and they realize that its a broader field than they realized and find they enjoy it quite a bit. I believe that there is something in any field that someone would love, they just haven't found it yet. (Like not all art is watercolor or oils, but sculpture and architecture. My boyfriend hates painting, but loves pottery--something he never really knew about until I showed him.)