It sounds to me like you two are already in a "friends with benefits" situation. You two are friends who have sex on occasion, but there is no confirmed commitment to one another exclusively; it sounds like you two have never talked about dating or your status. Just because it seems like you are dating does not mean you are. That's something that needs to be explicitly talked about, even though it can be a hard topic.

You seem to have found yourself in a tough situation: you're having casual sex with this guy, but you want more with him. From the sound of it he, however, does not want more with you. The only way to resolve this issue is to talk with him about it and see if you stand. If your wants and expectations don't match up, though, it's not fair to try and continue the relationship. You will most likely never feel satisfied, for sex does not substitute romantic connection and you will always be more invested than him. I would suggest that, if it turns out you want different things, to end your sexual relationship. It might be painful, since you have such a special connection to him, but in the long run it is what's fair to both of you.