Basically, I have this really annoying little fucker in my life who everytime he spots me interacting with people, says this. I’ve interacted with him very few times, so he basically doesn’t even know me.
While I am a little introverted, no one likes this bs.
Also, if you do decide you need to escalate past this asking them to explain the joke can work really well. It works wonders for bigotry and other issues too, if they say “it’s just a joke” say “I don’t understand, can you please explain the joke?” This makes them articulate their explanation for why they find it funny and chances are they will sound like a complete dingus in front of you and others, heavily disincentivising this behaviour.