No it isn't permanent. It's only cut in half when you hold the item. On the plus side you get more Attack EVs.
EV-enhancing items are items which enhance the EVs gained when a Pokémon is defeated. All of them halve the Speed of the Pokémon holding them in battle while they are being held. ~ Bulbapedia
Source: Experience