Flight always ends after a certain amount of time, but before you totally stop you will start to float straight down. You can shorten your flight time by pressing B again or touching onto ground. However, from your question, it looks like you may not be trying to move while you're flying, you can move like normal when you fly.
Source:experience