My husband had the same fears as you!
His real father was never in the picture and the man his mom has been with well... he's a twit.
I'm pretty sure my husband still has those same fears... I don't think they'll ever fully go away...
The best thing you can do is let your wife know about your fears, and tell her you need her support.. I'm constantly reassuring my husband that he's a wonderful father to our son, and that our son is very lucky to have him as his Daddy.
I'm sure your wife will do the same, you just have to let her in and let her know when those fears creep into your head.
congrats on the little fetus bean.