In plans outlined on Tuesday the regulator said the cap, set at 37%, will restrict BT-owned EE and Vodafone in the auction process, due to take place later this year.
Philip Marnick, Ofcom's spectrum director, said: "We've designed this auction to ensure that people and businesses continue to benefit from strong competition for mobile services."
However, smaller operators such as Three and O2 wanted the cap to be set at between 30% and 35%.
Three chief executive Dave Dyson described Ofcom's proposal as a "kick in the teeth" for consumers.
He said: "By making decisions that increase the dominance of the largest operators, Ofcom is damaging competition, restricting choice and pushing prices up for the very consumers that it is meant to protect.
"The mobile market is imbalanced and failing customers. Ofcom has shown little interest in tackling the problem. We will consider our response as a matter of urgency."
Smaller mobile networks like Three are concerned that the auction process will allow larger players to increase their dominance of the sector.
BT/EE holds 45% of all usable mobile spectrum, Vodafone has 28%, O2 15% and Three 12%.
However, Ofcom said that Three's consultation response " presented an overly pessimistic view of its own position".
An Ofcom spokesman said: "We take all our decisions in the interests of consumers.
"This auction will keep the airwaves fair by reducing the share held by the largest operator.
"It will include strong safeguards to maintain a healthy four-player market and allow mobile operators to acquire the airwaves they need to compete."
Ofcom's cap means BT/EE will not be able to bid for spectrum in the 2.3GHz band and Vodafone can bid for a maximum 160 MHz of spectrum across both the 2.3GHz and 3.4GHz bands.
EE boss Marc Allera said: "While we don't agree that spectrum caps were necessary for this auction, our focus remains on investing in our network, using our existing and future spectrum to provide the best mobile experience for our customers across the UK.
"We look forward to bidding for additional spectrum in this auction."