Nope The effect only last as long as the Pokemon holds the item. and it only activates in battle. The second you remove it he will stop losing his speed in battle. Source: http://bulbapedia.bulbagarden.net/wiki/EV-enhancing_item
