International adoption, interracial kids--some questions

I'm going to have to be un-PC in this post mostly so as to be understood; before I go any further, please let us agree that there's a vast gray area between un-PC and racist (which I clearly will avoid). Anyway, here goes.
Recently in Parenting magazine, a mother had an article about herself and her daughter. Mom is white, husband is Japanese, daughter is (obviously) a mix. When Mom is out with her little girl, she is overwhelmed by those presuming the little girl is adopted. "Where did you get her?" "Where is she from?" et cetera. She doesn't quite know how to reply and finds the questions a little rude. [I personally like "My husband is Japanese; she gets her features from him."]
One example from my own life is my OB. She's Asian (Chinese extraction?) and her husband is white. I wonder if he's asked the same questions when he's out with one or more of their four sons. I doubt many would have the nerve to ask.
Another is a friend who's Hispanic. Her two kids don't look Hispanic at all; though her son has some of her features, her daughter is blonde with blue eyes. I wonder sometimes if strangers in the grocery store think she's the baby sitter; ironically, she did come to the U.S. as an au pair.
I'll acknowledge international adoption--adoption in general, really--is a very good thing. Madonna's might be the exception. It's certainly more common and more acceptable than it was a generation ago, which is fine.
Neighbor Shelly was a bit offended by a hairdresser recently on this issue. Shelly has five boys. The cosmetologist had the stones to ask, "Did you keep trying for a girl?"
Shelly, the soft-spoken shrinking violet that she is, shot back, "Well, some of them are adopted, so I guess we got exactly what we wanted."
I can't help but play the statistics, though. While international adoptions are up, so is interracial marriage (from .2% to something like 5% since 1970). I have to assume that the vast majority of children are NOT adopted and are the biological children of the person watching over them regardless of the racial appearance of the kids. That's simply how I operate.

So I was a little dismayed by a conversation I had not too long ago at swim class. Dad was there with his two kids; he was perhaps Italian and the kids were Asian. We were chatting about our kids, which ones were respectively ours, and their proficiency in the water; I commented, "They must look like their mother."
He looked at me like I was stupid. "They're adopted." Almost contemptuous.
Frankly, that killed the conversation. Was I off? It's not like it's unheard of for a white guy to marry an Asian girl. I suppose I'd like advice on this; what's the scoop? What are your thoughts, Gentle Readers?