It's good to have standards so you know what you are looking for in a partner, and looking for someone you can carry a conversation with is definitely a good standard to have. However, you may be limiting yourself if you only take interest in women involved in mathematics or who are familiar with it. Try exploring things with women who have different passions than you, if only to learn more about them. It doesn't have to be a permanent thing or a long term relationship, just go out on a date or two and see what you think; give it a chance. It's good to be able to carry conversation in more than one thing, anyway; it makes you a more well-rounded individual. See if these girls can help you with that.