10/27/2011 10:48 am ET | Updated Dec 27, 2011

Bernie Madoff: Worst Part Of Prison Is Knowing My Family 'Hates Me'

Bernard Madoff -- the man convicted of swindling billions from clients worldwide -- said he's more concerned with how his crimes affected his family than the victims of his scheme.

"I can understand why clients hate me," Madoff told ABC's Barbara Walters in an interview in prison. "The gravy train is over. I can live with that."

But the worst thing about his new life in a North Carolina prison is "Not seeing my family and knowing they hate me," he told Walters. "I betrayed them."

Madoff, who is currently serving a 150-year prison sentence, was arrested in December 2008 for running the largest recorded Ponzi scheme in history, which cost his victims his $64.8 billion in paper losses and $18 billion in cash losses, according to The New York Times. The victims ranged from charities to university endowment funds to more traditional investors.

Madoff has tried to defend himself at times. In February, he told New York magazine he essentially considered himself a "good person."

The Walters interview, which lasted two hours and wasn't on camera, is one of many media appearances from the Madoff family in recent weeks. Ruth Madoff, Bernie's wife, told CBS's 60 Minutes and the NYT that she and her husband attempted suicide on Christmas Eve in 2008, shortly after Madoff was arrested. The couple took sleeping pills, but woke up the next morning, according to the reports.

Earlier this month, Madoff's daughter-in-law, Stephanie Madoff Mack told ABC News that she blamed Bernie Madoff for her husband Mark Madoff's suicide in December 2010. Mark Madoff, Bernie Madoff's oldest son, also attempted suicide in 2009, Mack told ABC.

"I hate Bernie Madoff," Mack told ABC. "If I saw Bernie Madoff right now, I would tell him that I hold him fully responsible for killing my husband, and I'd spit in his face."

Irving Picard, the liquidator of Madoff's firm, has sued JPMorgan for $19 billion lawsuit in an attempt to win back some of the losses of Madoff's victims, according to Bloomberg.