There isn't a complex formula for Iron Fist boosts, it's a simple 20% increase to punching moves. No calculation required at all.
As for the order they're calculated, it does not matter one bit. It does not matter which order you put them in, you're going to have the same outcome, regardless of what numbers are used. For example:
20 x 1.5 (STAB) = 30. 30 x 1.2 (Iron Fist) = 36.
20 x 1.2 (Iron Fist) = 24. 24 x 1.5 (STAB) = 36.
If you really care though, STAB goes first according to: http://cdn.bulbagarden.net/upload/3/32/ModifierCalc.png