If the Obama administration's mortgage-relief program has been a bit of a flop, it didn't have to be so.

In a new book, TARP's former inspector general claims that he warned Timothy Geithner's Treasury Department repeatedly that the mortgage program, HAMP, was a disaster waiting to happen. Instead of listening, he says, Geithner plowed right ahead with it, to serve the banks.

Neil Barofsky, the congressionally appointed watchdog for the Troubled Asset Relief Program, which pumped $700 billion to banks, auto makers and homeowners after the crisis, argues in the book that the Home Affordable Modification Program introduced in early 2009 was poorly thought out and executed, opening the door for abuse.

"The hurried rollout of HAMP would soon bring with it a rash of misconduct and criminal activity," Barofsky writes. "Treasury's bungling of HAMP and its refusal to heed our warnings and those of other TARP oversight bodies resulted in the program harming many of the people it was supposed to help."

As it turned out, the main beneficiaries of the mortgage program were the banks -- a repeated theme in Barofsky's book, "Bailout: An Inside Account Of How Washington Abandoned Main Street While Rescuing Wall Street," a copy of which was obtained before its release by The Huffington Post. The book is fairly blistering in its assessment of the Treasury Department's handling of the bailouts, saying Geithner & Co. consistently took pains to avoid causing problems for the banks, often at the expense of homeowners and taxpayers.

"We haven't seen the book, but we wish Mr. Barofsky well," said Treasury spokesman Matt Anderson, in response to a request for comment.

Even without Barofsky's charges, HAMP has widely been viewed as disappointing. It has given permanent relief to nearly 1 million homeowners, but that is well shy of the 3 to 4 million it was intended to help. More borrowers have had their modifications canceled than are still in the program. The program has been riddled with some of the servicer abuse and incompetence Barofsky says he saw coming.

In fact, a new Government Accountability Office report about TARP released on Thursday agrees with many of Barofsky's assessments of HAMP. The program hasn't reached nearly as many people as intended, the GAO found, and Treasury has failed to properly assess risks or ensure transparency in the program.

"Treasury may have difficulty mitigating potential risks, such as an increase in redefaults or the misuse of funds; effectively assessing program outcomes; or holding servicers accountable," the GAO wrote in its report.

Barofsky's concerns about HAMP began when he watched a "twitchy, sweaty and obviously nervous" Geithner announce only bare-bones information about the program, as part of a broader new bailout plan, in February 2009 -- a dismal performance by Geithner that caused the stock market to tumble that day.

"The markets were hoping for detailed programs, and they expressed their disappointment with Geithner's vacuous speech with a large sell-off," writes Barofsky, 42, a former Manhattan federal prosecutor. A self-described Democrat who donated to President Obama's 2008 campaign, he was appointed by President George W. Bush.

Further details of the mortgage plan were painfully slow in coming.

"That became Treasury's modus operandi," writes Barofsky. "[F]irst, announcements intended to 'shock and awe' the media that made for good sound bites but were not particularly well thought out; then, weeks later, scattered and incomplete details that had to be reworked on the fly. And finally, poor program execution that accomplished little, if any, of the originally announced goals."

The few details that did come to light troubled Barofsky. The program was essentially going to be operated by mortgage servicers, who were not prepared to handle millions of modifications. They also had no incentive to help homeowners, according to Barofsky, because in a foreclosure they get their fees before anybody else. Barofsky worried this conflict of interest would "cripple" the mortgage plan.

He also worried that the program did not include "tight underwriting standards, such as requiring written verification of borrowers' residence and income." He thought mortgage servicers would "try to collect payments for loans that either did not exist or that were so deeply in default that they had no chance of qualifying for the program." And he feared homeowners could fall prey to "fraudsters, who would charge struggling borrowers large up-front fees in return for empty promises" of mortgage help.

On Feb. 26, 2009, Barofsky says, after getting no response from Treasury officials about his concerns, he sent Treasury official Neel Kashkari a set of recommendations for protecting homeowners. Kashkari, another holdover from the Bush administration, said Treasury considered itself advised. "Several" of those recommendations weren't included, however, and Barofsky says the fraud soon followed.

"After just a few months, we were already seeing a rash of just the kinds of abuses we had feared," he writes.

In response to homeowner complaints about mortgage servicers, Treasury "demonstrated no interest in taking even the most modest steps to punish them," Barofsky writes. "That was unconscionable, given the pain being inflicted on so many home owners."

In a meeting with Geithner -- this one involving fewer f-bombs than others -- Barofsky says he finally realized the root of the Treasury Department's apparent lack of interest in helping homeowners: They apparently had another goal in mind.

At the meeting, Elizabeth Warren, then chair of a congressional oversight panel established in 2008 to oversee the bailouts, questioned Geithner about HAMP's ability to help homeowners -- not the last time she would grill him.

"In defense of the program, Geithner finally blurted out, 'We estimate that they can handle ten million foreclosures, over time,' referring to the banks. 'This program will help foam the runway for them.'"

To Barofsky it seemed that Geithner saw HAMP mainly as a way to stretch out the foreclosure process, giving banks time to recover from the crisis without having to be hit with a wave of foreclosures all at once.

"Helping the banks, not home owners, did in fact seem to be Treasury's biggest concern," Barofsky writes. "HAMP was not separate from the bank bailouts; it was an essential part of them."