Liverpool fortuitously took the lead when Ryan Bertrand deflected Steven Gerrard's cross into his own net.
In second-half injury time Reading were rewarded for their endeavour throughout when Gylfi Sigurdsson levelled from the spot after a foul on substitute Long.
Long's extra-time header proved the decisive blow to send Reading through.
This was Reading's first victory at Anfield and it was fully deserved as they out-worked their under-par Premier League opponents to book a fourth-round tie at home to Burnley.
The result places even more pressure on under-fire Reds boss Rafael Benitez who will also be concerned that star duo Fernando Torres and Steven Gerrard failed to finish the match due to injury.
Neither side had played since the original third-round tie; with Reading's home game against Newcastle and Liverpool's match with Tottenham both falling foul of the weather.
The team line-ups for the replay reflected the level of satisfaction each had taken from the game at the Madejski.
Reading caretaker manager Brian McDermott named an unchanged side from what was arguably their best performance of what has been a lacklustre season so far and he was rewarded with a similarly committed display from his side.
They looked anything but a team that has failed to win any of their last five league matches and have spent all season in the bottom eight of the Championship.
In contrast, Liverpool made five changes in an attempt to improve upon what was a blunted, mediocre showing in the 1-1 draw.
They started well and were only denied what could have been the game's opening goal when Ivar Ingimarsson's outstretched boot intercepted Gerrard's cross with Yossi Benayoun lurking.
But once the initial storm from the home side was weathered, Reading's confidence and assurance grew as Liverpool's wilted.
With a platform built, the visitors twice could have taken the lead either side of the 20-minute mark.
Firstly, Reds defender Jamie Carragher opted to leave Simon Church's right-wing cross, unaware that Grzegor Rasiak was lurking behind him, but the striker miscued his effort over the bar.
Then, Jobi McAnuff broke away down the right
before delivering into the box from deep but the cross was inches ahead of the diving Church who failed to make contact.Reading were further buoyed on the half hour when Torres was substituted after falling awkwardly and replaced by David Ngog.
Despite the loss of the prolific Spanish marksman and the deficiencies that have hampered their season thus far, Liverpool have the quality to exploit the slightest of lapses from an opponent.
On the stroke of half time they did just that.
Gerrard's initial corner was headed clear but it was returned to the England midfielder, who played a one-two with Ngog before directing a ball into the six-yard box which deflected off Bertrand and in.
Liverpool's joy was tempered by the withdrawal of the Gerrard at half time with a twinge in his hamstring.
However, despite the loss of their influential captain, Liverpool began the second half well and pinned the visitors into their own half for large spells.
But despite their dominance of possession they crafted relatively few clear goalscoring chances.
Daniel Agger flashed a 25-yard shot well wide after Alberto Aquilani's free-kick had deflected to him off the Reading wall and the Italian headed a good chance wide from a Benayoun cross.
In contrast, Reading created a number of good opportunities during the half.
Substitute Brian Howard swung over a superb cross to the edge of the six-yard box but somehow Ingimarsson failed to make contact with the goal at his mercy.
Not long after, McAnuff picked up the ball on the halfway line and drove at the Liverpo
ol defence, evading a string of challenges to find himself clear on goal but with the hard work done and keeper Diego Cavalieri committed to ground early, he shot wide of the upright.Reading's final chance appeared to have gone when Cavalieri palmed away a shot from Sigurdsson in the 90th minute but there was still time for another twist.
A long ball forward broke fortuitously to substitute Long in the Liverpool box and whilst Benayoun's tackle on the forward was mistimed more than malicious, no contact was made with the ball and referee Philip Dowd pointed to the spot.
Sigurdsson showed an admirable lack of nerves to side-foot the penalty in, with Cavalieri diving the wrong way.
Having rescued themselves at the death, a buoyed Reading stunned Anfield by taking the lead 10 minutes into extra-time.
Brynjar Gunnarsson nutmegged Emiliano Insua on the right before sending over a pinpoint cross that Long glanced into the Liverpool net from six yards.
Inevitably, the rest of extra time was played largely in the Reading half but despite their best efforts Liverpool were unable to force an equaliser.
Benayoun spurned a glorious chance to equalise when through on goal but he dwelt on the ball and Federici was able to save his shot and Ngog fluffed a good shooting opportunity after finding space in the box.
