IBM's Jeopardy machine was so dominant -- until the very end of the second day of the man-machine match, when it made what looked like a clueless mistake. It suggested Toronto as a "US city." And now instead of bowing before the new model of machine intelligence, masses of Jeopardy fans are ridiculing it on Twitter and elsewhere.
Just what IBM hoped to avoid. As I write in Final Jeopardy, the team building the machine actually had a group--the so-called "dumb team" -- to try to steer Watson away from embarrassing gaffes. These were most likely to occur in Final Jeopardy, where the clues are more complex and Watson is compelled to respond, even if it has low confidence, as it did in Toronto. The team even considered programming to the computer to throw up its hands when puzzled, and just say it didn't know. But instead, they let it guess.
Read the comments on Twitter, and there's lots of misunderstanding about how Watson works. Watson doesn't have lists of things it "knows." Every clue is a research project, and it comes up with the statistically most promising answer.
Here's a summary of the issues:
1) Watson can never be sure of anything. Is it possible that the old rock star Alice Cooper is a man? If Watson finds enough evidence, it will bet on it--even though the name "Alice" is sure to create a lot of doubt. This flexibility in its thinking can save Watson from gaffes--but also lead to a few.
2) Category titles cannot be trusted. I blogged about this earlier, in a post How Watson Thinks. It has learned through exhaustive statistical analysis that many clues do not jibe with categories. A category about US novelists, for example, can ask about J.D. Salinger's masterpiece. Catcher in the Rye is a novel, not a novelist! These things happen time and again, and Watson notices. So it pays scant attention to the categories.
3) If this had been a normal Jeopardy clue, Watson would not have buzzed. It had only 14% confidence in Toronto (whose Pearson airport is named for a man who was active in World War One), and 11% in Chicago. Watson simply did not come up with the answer, and Toronto was its guess. (It communicated its low confidence by adding a lot of question marks.)
Even so, how could it guess that Toronto was an American city? Here we come to the weakness of statistical analysis. While searching through data, it notices that the United States is often called America. Toronto is a North American city. Its baseball team, the Blue Jays, plays in the American League. If Watson happened to study the itinerary of my The Numerati book tour, it included a host of American cities, from Philadelphia and Pittsburgh, to Seattle, San Francisco, and Toronto. In documents like that, people often don't stop to note for inquiring computers that Toronto actually shouldn't be placed in the group.
Long story short: Watson screwed up on the clue. It comes up with a clunker or two in nearly every game. But it also gets lots of clues right -- and is close to being the greatest Jeopardy player ever.
Follow Stephen Baker on Twitter: www.twitter.com/stevebaker
Clay Farris Naff: In the Watson Era, Will the Computer Be Servant, Master, or Savior?
Charlie
Did Watson activate the buzzer solenoid or was that left to its avatar?
The avatar gives a bit of information about the state of the search/map/reduce/analysis processes. the little "orbits" change color to indicate the system's confidence (low -> red, high -> green), and the system shows how busy it is (could it be load average?) by how many "orbits" it illuminates.
I know there was a solenoid pushing the button. What isn't clear is whether the post-question light was sent into the server room which sent the solenoid activation or...
Did Watson arm a mechanism on the avatar which activated the solenoid when the post-question light was detected?
Hmmm... but it should have also noticed millions of other data signals that clearly indicate Toronto is a major city in Canada.
By the way, I didn't see the show but I'm assuming the humans didn't have access to a live database (e.g. internet) while they played? This would be a more relevant comparison in my opinion. I'll bet humans with access to the same database, combined with their judgement, would have won.
Because Canada is part of North America.
Because it is a Continentalist.
Yes, Toronto is not an "US" city...but it is a (North) American city.
And it's in our closest neighbor, and it's within 100 miles of the border (like 90 percent of Canada's population).
It's not like Watson said "Shanghai".
And then, there's the thought...well, maybe it "could be" a U.S. city...say if North America ever forms a "Union" like the EU.
Toronto's airport is named for a fellow named Pearson active in the First World War. But it's not Pearson's service in WWI that earned him notoriety.
That fellow goes by the name Lester Bowles Pearson (or, more commonly, Lester B. Pearson). He long-standing Canadian Minister of Foreign Affairs, he is also a former Canadian Prime Minister, winner of the Nobel Peace Prize for the creation of the United Nations Blue Helmets during the Suez Crisis, who, though he always had a minority government, established the national universal health care, the Canadian flag, the Canada Pension Plan, the official bilingualism of the country, the refusal to send troops in the Vietnam war...A truly remarkable man in Canadian history.
That aside, a computer can't feel a gut feeling and can't be perfect. It probably saw that a US city was named Toronto somewhere, but that nobody knew it but knew the Canadian Toronto. So it probably felt that Toronto was a legitimate answer. For instance, there is Vancouver, BC, Canada and a small US city named Vancouver in Wash St. On a side note, so many American cities are named after European ones...Milan, Athens, Paris...and those are just in the state of New York !
-An American born in Toronto.
To me Watson is mind-blowingly impressive. Those who poo poo likely have no reference point to evaluate against. Still, denigrating from ignorance is plain irritating.