The ballooning interest rate manipulation scandal at Barclays, coupled with stock market instability, is likely to fuel fresh doubts about the integrity of the stock market, insiders said.

“Every time people begin to gain a little confidence, something else comes up,” said Randy Frederick, managing director of active trading and derivatives at Charles Schwab. “If it’s not Europe, it’s [troubled] IPOs, or JPMorgan or Barclays. Something new blows up and people say, ‘I knew it was rigged.’"

Frederick said the string of banking scandals and market instability has led some investors to pull their money out of the markets entirely, contributing to the long-term decline in trading volume. Frederick predicted options trading volume will be down 5 percent to 10 percent for the year.

In recent weeks, federal regulators and market executives have joined the alarm that retail investors are losing faith in the integrity of the markets. New data suggests those fears are well-founded.

A string of Wall Street crises, including the 2008 stock market crash, the collapse of the mortgage market, the botched Facebook IPO and the scandals at JPMorgan Chase and Barclays have meant “some very heavy body blows experienced by the public,” Richard Grasso, former chairman of the New York Stock Exchange, told CNBC’s Maria Bartiromo on Tuesday. “It's been a real tough time for consumers who want to get back into the market."

Grasso’s comments followed remarks last week by Securities and Exchange Commission Chairman Mary Schapiro that investors have a "concern about the integrity of the marketplace." Schapiro told a congressional subcommittee that U.S. markets are threatened with "an unwillingness [on the part of investors] to ever engage in the markets again."

Investors are unsure "whether they're getting accurate and honest information" from companies looking to sell stock to the public and uncertain "whether the market structure itself is tilted against the individual investor and in favor of the institutional investor," Schapiro said.

A recent survey from financial research and advisory firm Tabb Group reported that 31 percent of investors had "weak" or "very weak" confidence in the stock market, compared with 15 percent in 2010.

“There are some people out there feeling like the game is rigged,” said Frederick. “There have been enough events to make them suspicious. I think that’s unfortunate. And the industry needs to continue to make efforts to allay the concerns.”