[Eternal]

I think it would really help if you went on another date with just him, but don't do anything sexual. Don't have your friend tell him that, this is between just you and him. It would be better if you told him yourself.
You two could go out to dinner and just talk and get to know each other. That will help you to feel a lot more comfortable when you're with him. You're right, not all hope is lost. It's going to take time to build up trust for him, so you'll just have to take these things slow. As for touching, start with just kissing and slowly work your way to more intimate things. What's most important above all is to communicate. Before you take the next step, talk to him and let him know you feel ready. If you get in the moment and find you really aren't, then be sure to speak up. If he is a decent guy, he will understand and slow down.