I have the same problem, and I think it has to do with not driving; I tend to navigate by landmarks and the look of the street instead of names and numbers (so much so that I'm really stumped sometimes on how to give directions to things I walk by every day. "Up that way and over a little ...")