Hey there,
I agree that you should talk to him, at least so that you can get some closure and either move on or have the issue dealt with. The conversation won't be as bad as you imagine it to be, so just try to breathe and be calm when you go find him for it. If he doesn't want to be your friend, then you don't need him in your life. Him pushing you away like this definitely isn't your fault.
Feel free to
PM me if you ever need to talk,
Kyra