IBM PREPARES COMPUTER TO TAKE THE ‘JEOPARDY!’ CHALLENGE
By JOHN MARKOFF
(ART ADV: Photo NYT 25 is being sent to NYT photo clients. Graphic is being sent to NYT graphics clients. Nonsubscribers can purchase one-time rights by calling: 1-888-603-1036 or 1-888-346-9867.)
c.2009 New York Times News Service
YORKTOWN HEIGHTS, N.Y. — This highly successful television quiz show is the latest challenge for artificial intelligence.
ADVERTISEMENT
What is "Jeopardy"?
That is correct.
IBM plans to announce on Monday that it is in the final stages of completing a computer program to compete against human contestants on "Jeopardy!". If the program beats the humans, the field of artificial intelligence will have made a leap forward.
IBM scientists previously devised a chess-playing program to run on a supercomputer called Deep Blue. That program beat the world champion Gary Kasparov in a controversial 1997 match (Kasparov called it unfair and secured a draw in a later match against another version of the program).
But chess is a game of limits, with pieces that have clearly defined powers. "Jeopardy!" requires a program with the suppleness to weigh an almost infinite range of relationships and to make subtle comparisons and interpretations. The software must interact with humans on their own terms, and fast.
Indeed, the creators of the system — which the company refers to as Watson, in a reference to the IBM founder, Thomas J. Watson Sr. — said they were not yet confident that their system would be able to compete successfully on the television show, on which human champions typically provide correct responses 85 percent of the time.
"The big goal is to get computers to be able to converse in human terms," said the leader of the team, David A. Ferrucci, an IBM artificial intelligence researcher, "and we’re not there yet."
The team is not aiming at a true thinking machine, but a new class of software that can "understand" human questions and respond to them correctly. Such a program would have enormous economic implications.
ADVERTISEMENT
Despite more than four decades of experimentation in artificial intelligence, scientists have made only modest progress until now toward building machines that can understand language and interact with humans.
The proposed contest is an effort by IBM to prove that its researchers can make significant technical progress by picking "grand challenges" like its early chess foray. The new bid is based on three years of work by a research team that has grown to 20 experts in fields like natural language processing, machine learning and information retrieval.
Under the rules of the match that the company has negotiated with the "Jeopardy!" producers, the computer will not have to emulate all human qualities. It will receive questions as electronic text. The human contestants will both see the text of each question and hear it spoken by the show’s host, Alex Trebek.
The computer will respond with a synthesized voice to answer questions and to choose followup categories. IBM researchers said they planned to move a Blue Gene supercomputer to Los Angeles for the contest. To approximate the dimensions of the challenge faced by the human contestants, the computer will not be connected to the Internet, but will make its answers based on text that it has "read," or processed and indexed, before the show.
There is some skepticism among researchers in the field about the effort.
"To me it seems more like a demonstration than a grand challenge," said Peter Norvig, a computer scientist who is director of research at Google. "This will explore lots of different capabilities, but it won’t change the way the field works."
The IBM researchers and "Jeopardy!" producers said they were considering what form their cybercontestant would take and what gender it would assume. One possibility would be to use an animated avatar that would appear on a computer display.
"We’ve only begun to talk about it," "said Harry Friedman, the executive producer of "Jeopardy!." "We all agree that it shouldn’t look like Robby the Robot."
ADVERTISEMENT
Friedman added that they were also thinking about who the human contestants should be and were considering inviting Ken Jennings, the legendary "Jeopardy!" contestant who won 74 consecutive times and collected $2.52 million in 2004.
IBM will not reveal precisely how large the system’s internal database would be. The actual amount of information could be a significant fraction of the World Wide Web now indexed by Google, but artificial intelligence researchers said that having access to more information would not be the most significant key to improving the system’s performance.
Eric Nyberg, a computer scientist at Carnegie Mellon University, is collaborating with IBM on research to devise computing systems capable of answering questions that are not limited to specific topics. The real difficulty, Nyberg said, is not searching a database but getting the computer to understand what it should be searching for.
The system must be able to deal with analogies, puns, double entendres and relationships like size and location, all at lightning speed.
In a demonstration match here at the IBM laboratory against two IBM researchers recently, Watson appeared to be both an aggressive and a competent contestant, but also made the occasional puzzling blunder.
For example, given the statement, "Bordered by Syria and Israel, this small country is only 135 miles long and 35 miles wide," Watson beat its human competitors by quickly answering, "What is Lebanon?"
Moments later, however, the program stumbled when it decided it had high confidence that a "sheet" is a fruit.
The way to deal with such problems, Ferrucci said, is not to build a fruit database, but to improve the program’s ability to understand the way "Jeopardy!" clues are offered. The complexity of the challenge is underscored by the subtlety involved in capturing the exact meaning of a spoken human sentence. For example, the sentence "I never said she stole my money" can have seven different meanings depending on which word is stressed.
ADVERTISEMENT
"We love those sentences," Nyberg said. "Those are the ones we talk about when we’re sitting around having beers after work."
