Montana Gov. Brian Schweitzer defended his remarks on Mitt Romney's Mormon heritage, saying his comment that Romney's father had grown up on a "polygamy commune" was taken the wrong way.

"People are taking this far away from where I was discussing," Schweitzer, a Democrat, said in an interview Monday with CNN. "I was saying that Mitt Romney currently has a problem with Latino voters. And it is ironic that his father had come from Mexico. You could think he could embrace his Latino roots."

Recent polls have shown Mitt Romney trailing President Barack Obama among Hispanic voters. The former Massachusetts governor has moved to increase his appeal to Latinos, including campaign appearances Monday with Sen. Marco Rubio (R-Fla.) a Hispanic popular with Tea Pary voters, and reaching out to Hispanic voters in states like Arizona.

Schweitzer told CNN he didn't intend his remarks as an attack on the Church of Jesus Christ of Latter-day Saints.

"I didn't say anything about religion," Schweitzer said. "The Mormon religion hasn't accepted polygamy in 120 years. As I said before, Mitt Romney, or his family that I know of, doesn't accept polygamy today."

The governor's original comments came during an interview last week with the Daily Beast. While discussing Romney's difficulty connecting with Hispanic voters, Schweitzer said it was "ironic" because "his family came from a polygamy commune in Mexico."

In that interview, Schweitzer clarified that he was "not alleging by any stretch that Romney is a polygamist and approves of [the] polygamy lifestyle."

Romney quickly responded to the Montana governor's comments.

"My dad's dad was not a polygamist," Romney said in an interview with Fox News. "My dad grew up in a family with a mom and a dad and a few brothers and one sister. They lived in Mexico and lived a very nice life there from what I understand and then when he was 5 or 6 years old there was a revolution in Mexico. They escaped."

Also on HuffPost: