What you describe actually has a name. It's called cuckolding, and it's not uncommon. I know many people who have this fantasy and engage in these types of scenarios and relationships. My question is, what confuses you about it?