What happens to Gerudo men?

We know Gerudo males are possible because of Gannon. We also know there are no voe in Gerudo town and that Gerudo women go out into the world to find a mate

I was concerned that all Gerudo were committing infanticide on their male children so I asked my kids about how this works. They said that males are rare and usually take the genetics of their father so are raised as Hylian.

I don't see any lore to support that though. What do you think? Is this explained somewhere?