Bettanewbie, Don't feel guilty, you've done a good thing to give him a loving home. We have a Chessie/Lab mix who has always had trouble meeting new people. Fortunately he has fewer issues with dogs. We were in the same situation and found that we avoided inviting people over because we didn't know how he would react. (Probably the wrong approach.) He was usually pretty good with people who showed absolutly no fear, but the least bit of heasitation could bring out the worst. So I sympathize completely, and commend your for rescuing your guy from a much worse fate.
If Casey is like our Chester he's the sweetest cuddliest thing one could imagine with the people he trusts. Just know that you've done a very good thing, even if Casey never completely socializes with people.