Well, it depends what rules you want to set. If you're playing between friends, then the rules might be more lax than other competitive battles.
Serebii gives a basic outline of the rules of the metagame.
The Smogon clauses and banlists will show you want moves you can/ can't use, what abilities are allowed, etc. (Note:These rules arnt official and you can choose to follow them or not.)
The most common clauses are:
Evasion ban: you cannot use evasion inducing moves (such as double team) or items (such as bright powder).
Moody ban: you cannot use a Pokemon with the ability Moody.
Species clause: you cannot have two of the same species of Pokemon, even if they are different form (e.g. Rotom-Was and Rotom-Heat are not allowed on the same team).
Sleep clause: only one of your opponent's Pokemon can be put to sleep at any one time by you (discounts Rest).
The Smogon link will give more indepth explanations.
EDIT: also, if you're not familiar with tiers, a tier is basically a life of Pokemon that are more fairly matched against each other, e.g. Lugia would make mincemeat of Serperior, so Lugia is in Uber, the highest tier, while Serperior is in NU (never used), the lowest tier.
