Ultimately, it is King Fergus who would have given his daughter's hand away. The reason the lords wanted to hear from Queen Elinor is because she acts as his mouthpiece.
The tournament is a social construct that derives its authority from two entities: (1) The king, whose approval is necessary for it to have power, and (2) an implicit social contract between the lords based on their collective approval of the system.
According to traditional practice, Lord Dingwall's son had won. When Merida challenged that tradition by invoking an unprecedented and questionable loophole, the lords looked to King Fergus (and Queen Elinor) for clarification. There were effectively three decisions they could make: (1) Reject Merida's participation in the tournament, forcing her to marry Lord Dingwall's son; (2) Accept Merida's participation in the tournament, permitting her to remain unmarried; or (3) Nullify the tournament as a decision-making process, allowing Merida to choose whatever she wanted. When the king and queen balked, the lords became frustrated, both for their stalling and questioning the social contract.
The third choice, to nullify the tournament, was the most radical and feminist one, but Brave's writers went even further: They had Merida make the decision, convincing the lords and her father to tear down their social construct and bestow the power it once held in the wisdom of their children.
More questions on movies: