Most of the federal bailout money pledged through the Hardest Hit Fund, a program meant to help ailing homeowners in states especially battered by the housing crisis, has gone unspent, according to a report released Wednesday by a government watchdog. And most of that was used for unemployment assistance, not refinancing mortgages.

"The program has had a stunningly slow start," Christy Romero, the inspector general of the watchdog agency with oversight of the Troubled Asset Relief Program, said in an interview with The Huffington Post. "They are still not getting money out to homeowners."

As of June 30, state housing agencies had drawn down just $1.1 billion of the $7.6 billion allocated for the program since 2010, according to the report. Of that sum, $351 million had been spent to assist just 43,580 homeowners -- mostly with unemployment aid, not home loan modifications.

The Hardest Hit Fund is a small part of TARP, which was established in 2009 with the aim of pumping $700 billion into the banking and housing markets in order to keep both from imploding. Since its inception, though, TARP has been beset by criticism that the Treasury Department, which oversees the program, has favored the interests of banks over homeowners. The most scathing critiques of the program have come from the Special Inspector General for the Troubled Asset Relief Program, or SIGTARP, the government watchdog appointed to oversee Treasury's handling of the program.

In a new book, Neil Barofsky, who stepped down as the inspector general of the program last year, claims that the hurried rollout of HAMP by Treasury Secretary Timothy Geithner led to "a rash of misconduct and criminal activity."

In response to homeowner complaints about mortgage servicers, Treasury "demonstrated no interest in taking even the most modest steps to punish them," Barofsky writes. "That was unconscionable, given the pain being inflicted on so many home owners."

Romero, who took over earlier this year, has also been an unstinting critic of Treasury's management of the program. Romero said that the Treasury Department has failed to set clear performance goals for homeowner assistance programs, and that it has also failed to force the banks and other loan servicers to treat homeowners fairly.

"If Treasury really is a steward of TARP it should be holding the [mortgage] servicers' feet to the fire," Romero said. "They need to be tough. I don't even see an acknowledgement from Treasury that what they are doing in housing is not good enough."

Andrea Risotto, a Treasury spokeswoman, said that Treasury has required participating loan servicers to "improve their processes" through ongoing reviews. These assessments, last released in June, show "clear progress by mortgage servicers," she said.

But only a small percentage of the money allocated to help homeowners has actually been spent. Of $45 billion in homeowner relief allocated by the Treasury Department, just $4.5 billion had been spent as of June 30, according to the inspector general report. Meanwhile, 407 banks and credit unions that were able to borrow hundreds of billions of dollars with little vetting still owe taxpayers more than $14 billion in principal.

The Treasury Department also said initially that its flagship Home Affordable Modification Program would benefit 3 million to 4 million homeowners -- but fewer than 1 million borrowers have had a mortgage permanently modified at a lower interest rate. Homeowners have complained bitterly about the application process, which for many was a numbing cycle of lost paperwork, endless phone calls with a series of poorly-trained bank employees, and ultimately in many cases, a rejection on grounds that they didn't understand.

The Hardest Hit Fund was intended to augment HAMP. It's not clear why some states have distributed more aid than others. Oregon, for example, has drawn down nearly half of the $220 million it was allocated, and has spent $50 million assisting 4,579 homeowners.

New Jersey, meanwhile, has drawn down $22 million of $300 million allocated -- and has spent less than $1 million of that. Just 171 borrowers have received assistance under the program, according to the report. A spokeswoman for the New Jersey Homekeeper Program, which administers the state's assistance fund, did not immediately respond to a request for comment.

It's also not clear what happened to the $800 million or so that states withdrew but did not spend. "We are not exactly sure what it is being used for," Romero said. Each state's program is administered separately, but it is likely that the money was spent on administration or overhead costs, or is being held in reserves -- pledged, but not yet doled out.

Risotto, the Treasury spokeswoman, said that states have "committed more than $722 million in program funds to help homeowners through the program, and continue to enroll additional homeowners every month."

"States are committed to spending all of their funds," she said.