Scouting basically means trying to reveal information about your opponent's team. In past generation, this can be Pokemon that the opponent has; however, the advent of team preview in Generation 5 means that is no longer necessary. Scouting nowadays mainly refers to trying to figure out what moves your opponent's Pokemon has. This is most commonly (and reliably) in the form of Protect; for example, Alomomola can use Protect in front of Volcarona to scout and see if it has Giga Drain or not.
You can use a Pokemon to bait the opponent into using a coverage move. For example, say I have Ting-Lu against the Kingambit, and I want to scout if it has Low Kick or not. I would switch into Moltres, who can take any of Kingambit's moves safely, and see what move Kingambit uses. If it uses Low Kick, that reveals the move, and Ting-Lu knows that Kingambit can threaten it. If it uses a different move, like Iron Head, Ting-Lu knows that it's safer against it.
There's also scouting for choice locks, and this is when you know for certain your opponent's Pokemon has a Choice item. This is, again, most commonly seen in the case of Protect. Using Protect Alomomola, a player can see if the opposing Choice Specs Iron Valiant uses Thunderbolt to try and KO Alomomola or Moonblast to predict a Raging Bolt switching in, for example. The player now knows that they can safely bring in a Fairy resist safely, such as Iron Crown, as the Iron Valiant cannot switch moves.
Finally, there's scouting for items, and this is rarer as few items can be scouted. Usually, the most common items that can be scouted are Choice items, as they give such a noticeable increase in stats that a player can usually notice them if in the correct situation, and having knowledge of a Choiced Pokemon means they are much more exploitable later on thanks to their restriction.
There's also scouting for EV spreads, and this is usually done with the help of a calculator. For Pokemon with a variety of common EV spreads, such as Landorus-T and Great Tusk, you can use the damage that they do to your Pokemon and/or damage that they take from your attacks to know the EV spread (assuming they're using standard spreads.)
As for how a player scouts, Protect is by and far the most reliable option, but switching into safe, defensive checks to scout is more applicable to most types of teams. As an example, Hyper Offense teams often don't have Protect; therefore, they have to leverage the natural bulk and typing of their Pokemon to scout moves, such as using Volcarona's defensive typing to scout if Rillaboom has Superpower or not for your Kingambit.
