Young Wall Street workers are getting fed up.

More than 20 percent of Wall Street employees born since 1980 say they would rather not work in the industry because of the blow its reputation took during the financial crisis, according to a survey from PriceWaterhouseCoopers, cited by Bloomberg. Nearly half of young Wall Street workers are actively looking for another job, the survey found.

These young adults, some of the nation's best and brightest, are rethinking their career choice at a time of near constant bank controversy. In one of the most notorious examples of Wall Street criticism, Greg Smith, a former Goldman Sachs executive, resigned from the firm via a March op-ed in The New York Times, in which he criticized Goldman’s environment as "toxic" in recent years.

He's not alone. Another young, former Goldman worker said the financial crisis took a toll not only on the larger economy, but on the company's reputation.

"You went from telling people quite proudly that you worked for Goldman Sachs, the number one investment bank in the world, to perhaps being a bit more cautious and saying you work in finance," Joe Nelson, who now owns his own custom-fit condom company, told Reuters in March.

Still, it may be more than just Wall Street's sinking reputation that's pushing younger workers to look for jobs elsewhere. Industry-wide pay cuts and layoffs have hit the youngest employees particularly hard. In January, Credit Suisse and other big banks mulled specifically freezing pay of their youngest workers.

In addition, the number of Wall Street workers between the ages of 20 and 34 dropped by 25 percent between the third quarter of 2008 and the third quarter of 2011, according to The New York Times. During the same period, Wall Street shrunk by 17 percent overall.

Add to the financial struggles that Wall Street's youngest are even more likely to be plagued by illness. A study of two dozen investment bankers found that every single one of them had suffered from a stress-related ailment like Crohn's disease, alcoholism or an eating disorder within six years of graduating from business school.

It's not only Wall Street's young workers that are suffering though. Wages for employees in their twenties nosedived over the past 10 years, according to a recent analysis from the Economic Policy Institute.