I accidentally pushed a boulder away, and now its blocking the way... Can I move them back? (I can't go behind and push it the other way because of those rocks)
Just exit the the cave room and re-enter—everything resets to where it was.