By Joseph Menn

San FRANCISCO (Reuters) - On March 9 of this year, a piece of Facebook software spotted something suspicious.

A man in his early thirties was chatting about sex with a 13-year-old South Florida girl and planned to meet her after middle-school classes the next day.

Facebook's extensive but little-discussed technology for scanning postings and chats for criminal activity automatically flagged the conversation for employees, who read it and quickly called police.

Officers took control of the teenager's computer and arrested the man the next day, said Special Agent Supervisor Jeffrey Duncan of the Florida Department of Law Enforcement. The alleged predator has pleaded not guilty to multiple charges of soliciting a minor.

"The manner and speed with which they contacted us gave us the ability to respond as soon as possible," said Duncan, one of a half-dozen law enforcement officials interviewed who praised Facebook for triggering inquiries.

Facebook is among the many companies that are embracing a combination of new technologies and human monitoring to thwart sex predators. Such efforts generally start with automated screening for inappropriate language and exchanges of personal information, and extend to using the records of convicted pedophiles' online chats to teach the software what to seek out.

Yet even though defensive techniques are now available and effective they can be expensive. They can also alienate some of a site's target audience -- especially teen users who expect more freedom of expression. While many top sites catering to young children are quite vigilant, the same can't be said for the burgeoning array of online options for the 13- to 18-year-old set.

"There are companies out there that are doing a very good job, working within the confines of what they have available," said Brooke Donahue, a supervisory special agent with an FBI team devoted to Internet predators and child pornography. "There are companies out there that are more concerned about profitability."

THE SMARTPHONE FACTOR

Two recent incidents are raising new questions about companies' willingness to invest in safety.

Last month the maker of a smartphone app called Skout, designed for flirtation with strangers in the same area, admitted its use had led to sexual assaults on three teenagers by adults. The venture-backed firm had not verified that users of its now-shuttered teen section were under 20, giving predators easy access.

Also in June, a teen-oriented virtual world called Habbo Hotel, which boasts hundreds of millions of registered users, temporarily blocked all chatting after UK television reported that two sex predators had found victims on the site and that a journalist posing as an 11-year-old girl was bombarded with explicit remarks and requests that she disrobe on webcam.

Former employees said site owner Sulake of Finland laid off many in-house workers earlier this year, leaving it unable to moderate 70 million lines of daily chat adequately. Sulake said it had kept 225 moderators and is still investigating what went wrong.

The failures at Skout and Habbo shocked child-safety experts and technology professionals, who fear they will lead to a renewed panic about online safety that is not justified by the data.

By some measures, Internet-related sex crimes against children have always been rare and are now falling (as are reports of assaults on minors that do not involve the Net). Most sex crimes against children are committed by people the children know, rather than strangers.

The National Center for Missing and Exploited Children processed 3,638 reports of online "enticement" of children by adults last year, down from 4,053 in 2010 and 5,759 in 2009.

Even those companies with state-of-the-art defenses spend far more time trying to stop online bullying and attempts to sneak profanity past automatic word filters than they do fending off sex predators.

Still, as the Skout case showed, there are several recent trends that have heightened the concerns of child-safety experts: the rise of smartphones, which are harder for parents to monitor; location-oriented services, which are the darling of Net companies seeking more ad revenue from local businesses; and the rapid proliferation in phone and tablet apps, which don't always make clear what data they are using and distributing.

EXPENSIVE DEFENSES

A solid system for defending against online predators requires both oversight by trained employees and intelligent software that not only searches for improper communication but also analyzes patterns of behavior, experts said.

The better software typically starts as a filter, blocking the exchange of abusive language and personal contact information such as email addresses, phone numbers and Skype login names. But instead of looking just at one set of messages it will examine whether a user has asked for contact information from dozens of people or tried to develop multiple deeper and potentially sexual relationship, a process known as grooming.

Companies can set the software to take many defensive steps automatically, including temporarily silencing those who are breaking rules or banning them permanently. As a result, many threats are eliminated without human intervention and moderators at the company are notified later.

Sites that operate with such software still should have one professional on safety patrol for every 2,000 users online at the same time, said Sacramento-based Metaverse Mod Squad, a moderating service. At that level the human side of the task entails "months and months of boredom followed by a few minutes of your hair on fire," said Metaverse Vice President Rich Weil.

Metaverse uses hundreds of employees and contractors to monitor websites for clients including virtual world Second Life, Time Warner's Warner Brothers and the PBS public television service.

Metaverse Chief Executive Amy Pritchard said that in five years her staff only intercepted something terrifying once, about a month ago, when a man on a discussion board for a major media company was asking for the email address of a young site user.

Software recognized that the same person had been making similar requests of others and flagged the account for Metaverse moderators. They called the media company, which then alerted authorities. Other sites aimed at kids agree that such crises are rarities.

NAUGHTY USERS, NICER REVENUES

Sites aimed at those under 13 are very different from those with large teen audiences.

Under a 1998 law known as COPPA, for the Children's Online Privacy Protection Act, sites directed at those 12 and under must have verified parental consent before collecting data on children. Some sites go much further: Disney's Club Penguin offers a choice of viewing either filtered chat that avoids blacklisted words or chats that contain only words that the company has pre-approved.

Filters and moderators are essential for a clean experience, said Claire Quinn, safety chief at a smaller site aimed at kids and young teens, WeeWorld. But the programs and people cost money and can depress ad rates.

"You might lose some of your naughty users, and if you lose traffic you might lose some of your revenue," Quinn said. "You have to be prepared to take a hit."

There is no legal or technical reason that companies with large teen audiences, like Facebook, or mainly teen users, such as Habbo, can't do the same thing as Disney and WeeWorld.

From a business perspective, however, there are powerful reasons not to be so restrictive, starting with teen expectations of more freedom of expression as they age. If they don't find it on one site, they will somewhere else.

The looser the filters, the more the need for the most sophisticated monitoring tools, like those employed at Facebook and those offered by independent companies such as the UK's Crisp Thinking, which works for Lego, Electronic Arts , and Sony Corp's <6758.T> online entertainment unit, among others.

In addition to blocking forbidden words and strings of digits that could represent phone numbers, Crisp assigns warning scores to chats based on multiple categories of information, including the use of profanity, personally identifying information and signs of grooming. Things like too many "unrequited" messages, or those that go unresponded to, also factor in, because they correlate with spamming or attempts to groom in quantity, as does analysis of the actual chats of convicted pedophiles.

The highest scores generate color-coded "tickets," with those marked red requiring the quickest response from moderators.

Facebook's software likewise depends on relationship analysis and archives of real chats that preceded sex assaults, Chief Security Officer Joe Sullivan told Reuters in the company's most expansive comments on the subject to date.

Like most of its peers, Facebook generally avoids discussing its safety practices to discourage scare stories, because it doesn't catch many wrongdoers, and to sidestep privacy concerns. Users could be unnerved about the extent to which their conversations are reviewed, at least by computer programs.

CATCHING ONE IN 10?

In part because of its massive size, Facebook relies more than some rivals on such technology.

"We've never wanted to set up an environment where we have employees looking at private communications, so it's really important that we use technology that has a very low false-positive rate," he said. In addition, Facebook doesn't probe deeply into what it thinks are pre-existing relationships.

A low rate of false positives, though, also means that many dangerous communications go undetected.

Some adults have used Facebook to target dozens of minors before assaulting one or more and then being identified by their victims or the victims' parents, court records show.

"I feel for every one we arrest, ten others get through the system," Florida's Duncan said of tips from Facebook and other companies.

Another pillar in Facebook's strategy is to limit how those under 18 can interact on the site and to make it harder for adults to find them. Minors don't show up in public searches, only friends of friends can send their Facebook messages, and only friends can chat with them.

The gaping hole in the defense of Facebook and many other sites popular with teens is that minors can easily make up a birth date and pretend to be adults -- and adults can pretend to be minors, as happened with Skout, which declined an interview request.

Technology is available for verifying the ages of Web and app users. One of the providers is Aristotle International Inc, which offers a variety of methods, including having a parent vouch for a child and make a token payment with a credit card to establish the parent's identity.

Yet even in the wake of the Skout disaster, no site aimed at minors has hired Aristotle for age checks. "We could do real parental consent with 14-year-olds, but no one has asked," said Aristotle Chief Executive John Phillips.

Such checks would cost money and alienate teens who don't want their parents involved.

Barring a wave of costly litigation or new laws, it is hard to see the protections getting much tougher, experts said. Instead, the app and location booms will only add to the market pressure for more freedom on youth sites and greater challenges for parents.

"For every Skout that shuts down, there are ten more that popped up yesterday," said the FBI's Donahue. "The free market pushes towards permissiveness."

(Editing by Jonathan Weber and Prudence Crowther)

Earlier on HuffPost:

Flip through the gallery to see some non-corporate alternatives to Facebook.
Loading Slideshow...
  • Pinterest

    Certainly the hottest new social network, Pinterest doesn't have all the functions and features of Facebook quite yet -- basically, you're just posting photos to your different boards, which you can categorize by interest or hobby or whatever. You can also follow your friends' boards and comment on their pins. And that's it. Pinterest is a simple, visual concept that has a huge, vibrant community of active users. It <a href="http://techcrunch.com/2012/02/07/pinterest-monthly-uniques/" target="_hplink">hit 10 million users faster</a> than any other social network and is now the <a href="http://www.huffingtonpost.com/2012/04/06/pinterest-traffic-growth_n_1408088.html" target="_hplink">third most popular social network</a> in America, trailing only Facebook and Twitter. <br> <br> You can <a href="http://pinterest.com/" target="_hplink">request an invitation to join Pinterest here</a>.

  • Tagged

    Tagged has a remarkably similar arc to Facebook: Also founded in 2004, and also originally tageted at young people, Tagged is now open to everyone and allows you to customize your profile, play games, message friends, post photos, and meet new people. It has more than 300 million users and more than twenty million monthly active users -- not too shabby, and perhaps worth a look if you want a robust Facebook alternative that's not going anywhere. To see what Tagged is all about, check out this video introduction for beginners. You can <a href="http://www.tagged.com/?" target="_hplink">sign up for Tagged here</a>.

  • Path

    Path is one of several new social networks that seeks to improve on Facebook by making the experience more private and personal: Users are limited to 150 friends on the mobile-only service. A user is instructed to only add his or her closest friends, or anyone you'd invite to your birthday party; the average Path user <a href="http://www.nytimes.com/2012/04/15/business/path-familyleaf-and-pair-small-by-design-social-networks.html" target="_hplink">has 40 connections</a>. Path is sort of like a daily online journal that you open to your friends: You can post photos and videos using your smartphone's camera, update your location, share what songs you're listening to and more. Path has been praised for its intimate feeling and clean design. For more on this social network and its mission statement, check out the introductory YouTube video. (Ignore the Thai -- it's in English). <br> <br> You must have an iPhone or Android phone to participate in Path; a BlackBerry app is apparently on its way. Path has about a million active users, per a <em>recent <em>New York Times</em> article</em>.

  • Pair

    Speaking of intimacy: Pair is a social network in which you can only have one connection, as its name implies. Pair is a sharing service for couples (or really good friends, I suppose), available on Android and iPhone. It takes privacy to the extreme: Pair calls itself a "timeline for just the two of you, where you can post cute video messages and photos that no one else will see." Your significant other may be forcing you to join it any day now. On Pair, you can share photos, videos, location, and to-do lists; you can also play Tic-Tac-Toe with one another and draw sketches in real-time. One of the most precious features of Pair is its "thumbprint" feature, on which you and your partner can virtually press your thumbs together. Like Pair as a whole, you will probably either find this adorable or schmaltzy. Path is available for free <a href="http://itunes.apple.com/us/app/pair/id503663173?mt=8" target="_hplink">in iTunes</a> and <a href="https://play.google.com/store/apps/details?id=com.tenthbit.juliet" target="_hplink">the Google Play store</a>.

  • Highlight

    The number of social networks based on your current location, and your proximity to other users of an mobile application, is on the rise. These "social-location-mobile" (SoLoMo) apps dominated the recent South by Southwest festival, and the app that got the most press was Highlight. Highlight is iPhone only, and the mobile app hooks up with your Facebook and notifies you when you are near a friend, or a friend of a friend, or another Highlight user with similar interests. You can view this person's Highlight profile, and if you're intrigued, you can message that person and perhaps make a new friend or connection. Highlight CEO Paul Davison explains the app to Anderson Cooper in the accompanying YouTube video. Highlight is <a href="http://itunes.apple.com/us/app/highlight/id441534409?mt=8" target="_hplink">available for free in iTunes</a>.

  • Circle

    Circle is, like Highlight, an app that tells you who's around you; unlike Highlight, it has a very pleasant design and lots of options for what information you share publicly and who can see you. You sign up for Circle with your Facebook account; the iPhone-only app shows you when Facebook friends are nearby, and also when friends of friends are close. You can choose to toggle on and off public visibility, if you don't want to be visible to friends of friends. Your profile shows your different Facebook networks (your college, high school, hometown, etc.) and you also have a mini-bio with your name, relationship status and interests. All of this can be toggled on and off as well. Circle is available <a href="http://itunes.apple.com/us/app/circle-whos-around-you/id488720081?mt=8" target="_hplink">on iTunes</a>.

  • Kismet

    Another social location app for your iPhone, Kismet shows you who's around and lets you chat with your nearby neighbors; it also allows users to check in on Foursquare and see which other Kismet users are at their location. Kismet boasts a nice map view, which allows you to see a broad view of other Kismet users around you; there's also an invitation feature that allows you to invite your friends and other users to meet up at a certain place and certain time. You can <a href="http://itunes.apple.com/us/app/kismet/id490929215?mt=8" target="_hplink">download Kismet for iPhone here</a>.

  • Ban.jo

    Our final SoLoMo app (and hopefully the last time I will ever have to write "SoLoMo" ever again) is Ban.jo, which differentiates itself by being available for iPhone AND Android AND on the web. Accessibility! Aside from cross-OS availability, Ban.jo is more of the same: See who's currently around you in list or map view, message nearby folks, check in and update LinkedIn, Twitter, Foursquare and Facebook. Ban.jo is also the only one of these apps publicizing the number of users it has: Its press kit claims that Ban.jo has over one million users worldwide in 185 countries. You can download Ban.jo <a href="http://itunes.apple.com/us/app/ban-jo/id417076117" target="_hplink">for the iPhone</a> or <a href="https://play.google.com/store/apps/details?id=com.banjo.android" target="_hplink">for Android</a>; you can also <a href="http://app.ban.jo/?__utma=18700074.216795228.1337200636.1337200636.1337200636.1&__utmb=18700074.4.10.1337200636&__utmc=18700074&__utmx=-&__utmz=18700074.1337200636.1.1.utmcsr=google|utmccn=(organic)|utmcmd=organic|utmctr=(not provided)&__utmv=-&__utmk=204321424" target="_hplink">try it out at Ban.jo's website</a>.

  • Nextdoor

    Nextdoor is a social network for neighbors and neighborhoods. You join with your home address and are immediately placed into a home neighborhood; all of your connections, and all the content you see in your feed, comes from those that live near you. You don't have to make your address visible to your neighbors, but you do have to verify that you live there with Nextdoor in order to use the site. After you join, using Nextdoor is like a mix of browsing Craigslist and using your community bulletin board. You can find out what's happening in your 'hood and get recommendations for different local businesses and services; there's also a classifieds section for buying and selling. You can check out Nextdoor's pitch in the accompanying video. You can <a href="http://nextdoor.com/" target="_hplink">sign up for Nextdoor for free here</a>.

  • Roamz

    A mobile app for Android and iPhone, Roamz brings in information from Twitter, Foursquare, Instagram and Facebook to let you know what cool stuff is happening around you -- "where the locals go," it claims. That's the real draw of Roamz. It's a social network where you can post status updates and photos and also get information about the places nearby. Check out a video for the app -- which its creators call "Social Googles for the Real World" -- on the left. You can download Roamz for <a href="http://itunes.apple.com/us/app/roamz/id459343660?mt=8" target="_hplink">free for iPhone</a> or <a href="https://play.google.com/store/apps/details?id=com.roamz.app&feature=search_result#?t=W251bGwsMSwxLDEsImNvbS5yb2Ftei5hcHAiXQ.." target="_hplink">for Android</a>.