外国人「日本語で話しかけてるのに日本人は引きつった顔で“I don’t speak English”って返すのなぁぜ？」

Why do a lot people in Japan say “I don’t speak English” to a foreigner who just spoke to them in Japanese?
One of my dad’s cousins (he’s white for context) works in Japan a lot, so he’s pretty close to
fluent in Japanese. He told us that if he walks up to someone and says in Japanese “Excuse me, where is the restroom?” they’ll say “I don’t speak English”. Then he’d be like “no, I speak
Japanese”, and the other person would sometimes still be like “I don’t speak English”.
And it’s not even a random person he walks up to most of the time, it’s a worker at the store
he’s in who you can assume wouldn’t be like “don’t talk to me”.
The other day I saw a whole thing on a language learning sub Reddit about this exact situation,
but no one in the comments gave an explanation for why this is so common. Beyond possible
dialect differences what’s the reason for this?
https://www.reddit.com/r/NoStupidQuestions/comments/196or5v/why_do_a_lot_people_in_japan_say_i_dont_speak/

て感じ？

多分EUが日本に宣戦布告したら俺のせい