Gardevoir's Attack is 60 points lower than its Special Attack, meaning you give up a lot of firepower if you choose to run a physical set. You're choosing to not run the best moves for the job: Gardevoir doesn't get Play Rough or physical coverage higher than 75 BP. Its special movepool is expansive and comes off a much higher attacking stat --- there's no reason to handicap yourself by using physical options.
Can it be done? Of course, but you'll find it's a lot more trouble than it's worth.