06/04/2012 01:42 pm ET

Hilary Duff: I'm Not Doing Anything 'Drastic' To Shed The Baby Weight

In Hollywood it often seems as if it's a race for new moms to shed the baby weight as fast as humanly possible, but two months after giving birth, Hilary Duff isn't rushing anything.

"I think if you ask any pregnant mom, they're like 'I want my body back,' but it takes time," she told Us Weekly at the Bing Summer of Doing charity event on June 1. "It takes nine months for your body to get that way and it's putting on that weight on purpose. The second I start to get down like 'what happened to my body,' I look at my beautiful baby and I've never been more appreciative for the body that I have."

The 24-year-old singer and actress has been spotted hitting the gym a few times a week in an effort to start shedding the pounds, but says she's not overdoing it.

"Obviously, I'm not where I want to be .... but I'm working hard and I'm doing everything I can to lose the weight the right way," she told the magazine. "I'm breast-feeding, so you have to be careful and not do anything too drastic."

Duff has a very different attitude from fellow new mom Beyonce, who pushed herself to lose 60 pounds just five months after giving birth.

"Y'all have no idea how hard I worked. I had to lose 60 pounds! They had me on that treadmill -- I ate lettuce!" she told the audience at her Memorial Day weekend show in Atlantic City -- her first concert since giving birth to Blue Ivy on Jan. 7.

As for Duff, she told Us Weekly she's been far more interested in baby Luca's every move.

"Okay, I'm going to brag for a second. We were at the doctor for one of his checkups and he rolled over," she told the magazine. "He's really only rolled over like four times, but he did it in front of [his] pediatrician, and [the doctor said], 'Wow, they usually start doing that at three months. That's pretty good.' I was like 'Yes!'"

Hilary Duff