"Hey, Guys!": Just a Fun Greeting, or a Derogatory Phrase?

Tonight my father brought something to my attention. The two of us, along with my mother and aunt, went out to eat at an Italian restaurant. He enjoyed himself and had a good meal, but one thing irked him: our waiter kept addressing us as "guys" and my father was the only man there. Nothing against the waiter, for he did a good job serving us, but my father found fault with this because he's a food writer and notices things like this. He felt that instead the waiter should have referred to us as "folks" to make the term more collective. At first I didn't see the big deal with this because "guys" is such a general term, and I think I have addressed both my female and male friends as "guys" from one time or another. I think I may have also addressed you, my faithful readers, as such once or twice as well.

But then my father told me something that somewhat altered my thinking. What if the table had three men and one woman, and the waiter addressed us as "ladies?" Then all of a sudden it becomes weird. Why is it that it is so normal for us to call groups of people regardless of gender "guys," which is regarded as a male term, but we would never call groups of people "ladies" unless they are women, which is regarded as a female term?

Being the feminist that I am, this weighs on me further. Is addressing everybody as "guys" the social norm because we still live in such a patriarchal society? Should I be offended by this? As a woman, when I use the word "guys," am I insulting myself? I say this because after thinking about this now, I am kind of considering "guys" a sexist term. I never really thought of the term "guys" as sexist that much, but I'm having second thoughts about using it now.

This idea reminds me of the lyrics to Madonna's song "What it Feels Like for a Girl":
Girls can wear jeans
And cut their hair short
Wear shirts and boots
'Cause it's OK to be a boy
But for a boy to look like a girl is degrading
'Cause you think that being a girl is degrading

Source

Is this the same thing? It's okay for us to call people "guys" but the title "ladies" or even "girls" is only reserved for females. Is it an insult to be called "girls" because it is degrading to be a girl and is it okay to call anybody "guys" because it is okay to be a "guy?"

Is the simple word "guys" symbolic of a bigger picture and we use it without even realizing it because we are so used to it being part of our culture?