Limited duels in Yu-Gi-Oh! Ultimate Masters World Championship 2006 are duels where you duel against special opponents that do not appear in free duels. Also, there are deck-bound and duel-bound conditions that the duel is started or played only by sticking to them (e.g. Stealth Bird where you're not allowed to attack or Blue-Eyes White Dragon where you can only use a certain Limited\Forbidden List) hence the Limits.
If a player happens to defy a Limited Duel's deck-bound condition, they'll be alerted of the existence of certain cards in their deck and the Duel won't start. While defying a Limited Duel's duel-bound condition results in instant loss of the Duel.
Unfortunately, there is a catch to those Duels: Your opponent doesn't Abide to the conditions, thus putting you at a great disadvantage in some cases (e.g. while dueling Stealth Bird your opponent can attack, forcing you to win with either Effect Damage only or with a special condition(Exodia, Destiny Board,...etc)). Some of the decks are made to counter some possible counter strategies or conditions (e.g. Morphing Jar which could Deck you out even after you satisfy the more-than-60-cards deck condition).