Is there any sort of formula or calculation that gives the amount of damage done by Return or Frustration?
Yes. These formulas determine the move's base power.
Return's formula is Friendship / 2.5, and Frustration's is (255 - Friendship) / 2.5.
http://bulbapedia.bulbagarden.net/wiki/Return_(move) http://bulbapedia.bulbagarden.net/wiki/Frustration_(move)