06/26/2015 10:38 am ET Updated Jun 26, 2016

A Few Confederate Flags Down, But Thousands of Trappings Remain

No portrait in any house had ever shocked me more. I recently drove through Mississippi, and stopped in a town known for its extensive pre-Civil War architecture. Plantation houses still stood with tiny outbuildings that guides called "the servants' quarters," but in fact housed slaves.

A local woman invited us into her home for cocktails. "I want to show you Southern Hospitality," she said. She had lived on the West Coast, and seemed open-minded. But a tall life-sized portrait hung prominently above her living room mantel -- a young man in a Confederate Uniform and with a sword.

"That's quite a portrait," I exclaimed, shocked.

"Oh, that's my son." she said proudly.

"But he's wearing a Confederate uniform!"

"Yes, that's what the young people wear at the ball every year." She seemed to feel that it was just for fun, so was ok.

But as we sat down, the painting loomed high over us, dominating the room and the house.

The next day, we visited the home of William Faulkner -- one of my heroes. An African-American student from the University of Mississippi, which owns the house, showed us around. He mentioned that his nearby high school has two separate proms every year -- one black, one white.

Two years ago, the students elected a black homecoming queen. An outcry erupted, and the school decided that she could remain the homecoming queen that year, but that the following year, a white one would have to be chosen.

"That's horrible," I said.

"That's just the way things are," he explained very matter-of-factly. "We know the rules. We don't date your daughters or go to your church on Sunday."

"Why don't you move elsewhere?"

"I don't know where I'd go. This is where I grew up."

Calls for removing the Confederate flag at the South Carolina state capitol and elsewhere, following Dylann Roof's horrific shooting of innocent African-Americans in their church, should certainly be applauded.

But the problems are much deeper. Underlying attitudes also need to change.

Removing the flags could help heal tensions at the moment, but ultimately be empty gestures. Flags are weighty symbols, but also mere pieces of cloth. We also all need to alter what lurks beneath. Racism persists in manifold subtle and not-so-subtle ways.

A few years ago, I visited a court room in rural Virginia -- near Manassas, where the Civil War battles of Bill Run were fought.

The judge sat on a raised platform, and above him hung a single portrait -- that of Robert E. Lee in a Confederate uniform. Lee was an extraordinary man, but in this white dominated area, standing high above the judge, his portrait sent a powerful message.

Some Southerners say they still want to fly the Confederate flag because it represents part of their history. They feel comfortable with their moral beliefs, arguing that these come from the Bible. But the New and Old Testaments also teach justice, charity and love. Not all past behaviors should necessarily be respected or upheld. Morally wrong past behaviors should not be sources of pride simply because they are historical.

Such symbols foster harms. Racial violence and police brutality against blacks, and discrimination continue. MIssissippi and South Carolina have among the states with worst health, educational and poverty in the country; and in these states, these disadvantages disproportionately affect African-Americans.

Thousands of Confederate flags still fly today from not only state capitols and schools, homes. Even if they were all removed, Confederate garb remains. Even if the uniforms were eliminated, images of Lee would probably endure.

Hence, we need to recognize and address these deeper attitudes inside us all.

Removal of the flag from state offices and license plates should be only the beginning, not the end. We need to work to be less placid in accepting inequalities that persist against various groups -- not only African- Americans, but women, gays and lesbians and others.

We should be careful that the response to Dylann Roof does not stop with a few flags alone.