Honestly, you can't "let" your boyfriend have anything, when all is said and done. The notion of needing permission for something from your partner denotes a sense of possession and control and that is not healthy in relationships of any kind. Your boyfriend is his own person. He can choose who he spends time with. In the end, people make their own choices, and we decide whether those choices are acceptable to us and adjust our responses accordingly.

I understand why you'd be hesitant, especially because you are so reticent around guys. But did your boyfriend ask you to do that, or is this just something you've done yourself? If your boyfriend asked you to do that that's really controlling and unhealthy behavior. But if it's just something you started doing on your own it's unfair to expect your boyfriend to do the same unless you've talked about it.

In the end, it's okay to have friends of the opposite sex. Media makes it a myth that your partner will automatically cheat on you. I would give him the benefit of the doubt, and maybe talk to him about your concerns.