I had a hard time adjusting when I first went away to college, too. My brother and sister-in-law didn't have too much trouble even though they don't speak Japanese. We were there for 2 weeks over Christmas. It was 10 years ago.
I studied Japanese for two years in college and visited Japan (Tokyo, Kyoto, and Osaka) for ten days. I am trying to study on my own for the JLPT next year. Unfortunately, I forgot a lot from lack of use, so I'm back to the basics.