It's totally random. Considering that none or more than one Monster House can appear on the same floor, there's no set chance of them happening.
There is the IQ ability House Avoider that will not cause your partner Pokemon (if it has this skill) to wander into a Monster House if you're separated, but that's about it for reducing chances of encountering a Monster House.
Monster House