I don't think there really is a problem with it. You'd be both adults. Maturity is another thing, however. I guess it just depends on the person. I think guys tend to date girls younger than them anyways (not all, but a lot).
Now, on the other hand, if you kept the age difference, but maybe took three years off both of you, in my eyes, it'd be a bit odd to have a 20-year-old date a 15-year-old (aside from it being illegal). But then again, it could be a mature 15-year-old.
I know my boyfriend liked & made-out with a couple of girls who are about 3-4 years younger than him. For him to be 20-years-old now, it doesn't seem to be that big of a deal since the girls are about 16-17. But at the time, he was like 18, making out with a 15-year-old or at the other time, he was 19 & liked/made out with a 15-year-old. To me, that's a bit disgusting & wrong since they were kids. Also, his sister is younger than me, but still older than those girls. I think it's weird he wanted to do that stuff with girls younger than his sister. I couldn't imagine dating someone more than a year younger than me, because my sister is three years younger than me & anyone younger would be a kid (kind of like your point about your "baby" sister).