Is it selfish of me to think it's selfish of them to say No to me?
Think about it. If I ask you to join me for dinner and you decline, is it selfish of you or me? I mean, who am I to think that you should change your every day plans to spend time with me? That's selfish, right?
Or is it?