Hey being little isn't a bad thing!
I'm barely 5 feet on a tall day. My boyfriend doesn't mind at all; in fact, he loves it. I enjoy it so much when he lifts me up and swings me around.
So that's not a factor at all! And if you really have strong feelings for this guy, I say go for it. Nothing's stopping you! And since you're only 15, there's not really anything to lose. Worst case is rejection... but so what? You'll get over it and move on in no time. Better than left wondering "what if?"