12/14/2012 05:18 pm ET

Facebook vs. Hackers: Why The Social Network No Longer Plays 'Whack-A-Mole'

Two years ago, Facebook's security team began noticing a surge in a particular strain of computer virus. The malicious software stole credit card information and other personal data belonging to thousands of Facebook users and spammed their friends with links to rogue websites, creating a global network of infected PCs.

At first, Facebook played defense, helping victims clean their computers and attempting to block each new strain of the malware known as "Yahos." But the virus kept mutating to avoid detection. So Facebook's researchers did something unusual for a business being hacked: They investigated the hackers themselves, placing the giant social network at the forefront of a more combative strategy in the war against cybercrime.

"We're not happy playing the whack-a-mole game," said Facebook security researcher Mark Hammel. "We like to be more aggressive and find the root cause of the problem."

That aggressive approach led to the FBI's announcement this week that authorities had arrested 10 members of an international cybercrime ring with Facebook's assistance. The suspected hackers -- who were not identified but came from around the world -- allegedly created a virus that infected more than 11 million computers globally and caused more than $850 million in losses.

For years, companies have purchased anti-virus software or hired security firms to protect them in an endless game of cat and mouse with cybercriminals. But hackers have become increasingly sophisticated, and Facebook believes that merely playing defense against their vast arsenal of hacking tools is no longer enough.

In this case, Hammel said, the company reverse-engineered the computer virus to identify who was spreading the malware. Reverse-engineering is part of Facebook's multifaceted security strategy, which also includes sending cease-and-desist letters, paying independent researchers to find security flaws on its site and filing lawsuits to win civil judgments against spammers.

Earlier this year, Facebook worked with outside security researchers to identify and publicly release the names, aliases and photographs of five suspected Russian cybercriminals accused of operating a virus known as "Koobface." It prompted Facebook users to install software to watch a video and thereby installed malware on their computers that helped cybercriminals to commit advertising fraud and steal from users' bank accounts. Facebook's efforts appeared to shut down the Koobface virus, although the hackers still haven't been arrested.

The Yahos malware created the Butterfly botnet, a network of infected computers that harvested bank account passwords and other personal data on millions of computers. Computer security experts say botnets contribute to the majority of malicious activity on the Internet today.

Facebook isn't the only major tech company stepping up its fight against hackers. Microsoft has also become more aggressive, filing lawsuits to obtain legal permission to shut down servers controlling botnets infecting millions of its users' PCs.

In the long term, Facebook believes it's more effective to focus on efforts that lead to the hackers' arrest, said Nick Bilogorskiy, a former security researcher at the company.

"Facebook's approach of identifying the people and putting them in jail is groundbreaking," said Bilogorskiy, who is now director of security research at the security firm Cyphort. "Nobody else has been able to do that so far."

Facebook declined to detail how it identified the cybercriminals arrested this week. But Bilogorskiy said common approaches include analyzing the malware's code for patterns that can be linked to specific hackers and locating "patient zero" -- the first person to have a computer infected by the virus -- who is often the hacker himself testing out the malware.

According to Hammel, the Butterfly botnet affected less than 1 percent of its 800 million users. Facebook said in a statement it has seen no new infections since October. But Facebook also cautioned that many computers may still be infected and directed users to a link to check for infections and receive free anti-virus software.

Bilogorskiy, who left Facebook last year, said he spent months fighting the Yahos botnet because the cybercriminals used multiple strains of malware to work around Facebook's defenses. He said Facebook and other companies can protect their networks in various ways, but there's only one way to really stop cybercriminals.

"The final strike -- the one they can't recover from -- is when law enforcement knocks on their door," he said.