I did a Google search, and apparently you are far from being the only person who has this problem. While I couldn't find anything released by GameFreak regarding how this feature works, it's apparent that they developed face-recognition software that uses the camera on the 3DS.
As you may have guessed, this software is not the best. It fails to take into account the diversity of people's faces and the different ways that people express emotions.
This mistake isn't exclusive to GameFreak, either. For instance, this video shows how early versions of HP's face recognition software worked much better with white people than people of other races.
TL;DR: It's pretty crappy software that works better for some people than for others.
I had the same problem myself. If you'd like some advice, show some teeth for the small smile and have your mouth slightly open for the big smile.
Unless you're able to have a different face, the best solution is to practice a lot (or get Sun/Moon, where you only need 3 Rainbow Beans for max affection).