It sounds like this guy really cares about you and accepts you for you, which is great. If you're concerned I would open up to him about your fears. Honesty and open communication make for good relationships, and I am sure you will find reassurance from him. Try not to stress too much, and trust in him.