I'm just curious, since all of the Pokemon on my team on my X Version have max affection.
Yes they wil only if you poke them and drop the pokepuffs instead of holding them near the Pokemon's mouth.
Other than these,Affection in Pokemon amie will never go down.
Source: Experience and http://gaming.stackexchange.com/questions/137873/does-a-pokemons-affection-level-in-pokemon-amie-decrease-over-time
Hope helped!