After falling behind to Kieron Freeman’s first-half strike and, despite Ben Gladwin’s penalty miss, Sam Ricketts drew Town level before Byrne struck late on.
Mark Cooper made nine changes to the starting line-up with the only notable absentee being Harry Toffolo who was ruled out through injury. Jermaine Hylton and Michael Smith were paired together as the strike partnership.
The hosts had the better of the early exchanges, roared on by a passionate home crowd, with chances from Harris and Davies not finding the target.
19 minutes into proceedings, United took the lead through Kieron Freeman. The defender took advantage of a shooting opportunity presented to him just inside the right-hand channel of the box to drill the Blades in front, beating Foderingham at his left-hand post.
Town were awarded a spot-kick around the half-hour mark when Nathan Byrne was felled in the box as he cut in off the right flank. Ben Gladwin took responsibility from the spot but was denied by a smart save from Mark Howard, who dived to his right to keep out the strike before recovering to claim the loose ball.
Despite the one-goal deficit, Town would create the better chances for the remainder of the half. Firstly, Byrne picked out Hylton in the area but he could only lift his shot over the bar. Massimo Luongo did likewise after being picked out by the full-back but couldn’t hit the target. In between the two efforts, a break led by Hylton ended with Byrne’s shot being blocked away by Howard.
The first chance of the second half fell to Swindon: Jordan Turnbull galloped forward from the back to join in an attack and clipped a cross in towards Michael Smith but the Town striker couldn’t direct his header on target.
Five minutes later, Town drew level through Sam Ricketts. Good work from Hylton forced a corner and from Byrne’s delivery, the on-loan Wolves man bulleted a header into the far corner of the net.
With just over ten minutes to play, Steven Davies spurned a glorious opportunity to hand the home side the advantage once again. A fizzing cross from the United right beat every Town defender and reached Davies at the far post. Applying the faintest of touches, he sent the ball marginally wide instead of into the back of the net.
As the clock ticked nearer 90, Jack Stephens set off on a run from the edge of his own box. His drive took him to the edge of the Blades’ six-yard box but he could only fire into the side-netting.
Then, deep into stoppage time, Nathan Byrne picked the ball up on the right-hand corner of the box and unleashed a shot goalwards that bounced in front of Howard, through the ‘keeper’s arms and nestled in the far corner of the net to spark scenes of unbridled joy among the travelling fans and on the Town bench.