There's an entire guide on how to EV train with Hordes, so yes, it is possible to get unwanted EVs if you're fighting the wrong type of Pokemon. You would never want to grind levels through Horde Battles when things like the Lumoise City restaurants and the Battle Chateau exist though, so this won't ever really be an issue.
If it's that big a deal, max out the EVs you want with Super Training before engaging in Horde Battles. As EVs don't matter during an in-game playthrough, and you can easily find better experience elsewhere, this will almost always be a waste of time.
