Today at Brown

James Hays has a goal that sounds relatively straightforward and can be simply stated: He wants computers to understand our world as well as we do.

Teaching a machine how to derive meaning from visual information is an almost impossibly complex task; the demands on machine memory are far beyond daunting. Yet it is a task any human toddler can easily master.

“Children are amazingly good at generalizing,” Hays said. “They see one quadruped that’s furry and they can generalize about dogs even though dogs come in all sizes and shapes. That’s way beyond computational capacity right now.”

Computers are increasingly able to decipher elements of our visual world. They recognize boundaries, geometric patterns, colors, textures. Given a sufficiently large database of images to sift through and compare — think tens of millions of amateur photographs from Flickr or Facebook — a computer can coarsely group what it sees into visual object categories. But that is only a start.

“There are some good estimates that we encounter 30,000 visual object categories in our life,” he said, “but could machines generalize from smaller corpuses of data? Do we need these massively data-driven methods? I’m of the viewpoint that yes — image understanding is very memory-intensive. You can’t take shortcuts.” You can, however, study what works.

“Looking at the best biological system for image understanding seems like an obvious thing to do when you’re developing artificial systems,” he said. “I’m not of the school that says you should try to recreate the brain to understand images, but the high-level insights and heuristics that the brain uses are probably transferable to the computational domain.”

Hays earned his bachelor’s degree at the Georgia Institute of Technology, summa cum laude, in 2003, and went on to earn his Ph.D. at Carnegie Mellon University last spring. He is currently in a postdoctoral program at the Massachusetts Institute of Technology and will begin his work at Brown in January 2010.

It was robotics — at least partly robotics — that drew him into computer science. Robots can be engineered to climb stairs, assist in delicate surgery, work in hazardous areas, simulate speech, vacuum floors, operate trains, weld autobodies — but their potential as interactive human assistants is limited by one major problem: Robots cannot understand the world and their place in it. Hays’s research on visual understanding may help define that key enabling technology, the best route to unlocking robotic potential.

“Robots are hobbled by the fact that they cannot understand the world. They’re kind of blind,” Hays said. “They have lots of other ways to interact with their surroundings — acoustics, range-finders, various sensors and so forth. But fundamentally, you need a robot to understand the world through the same methods we use.”

Hays is using computer graphics to get at that issue of equipping machines with a capacity to understand the world. There are colleagues at Brown who are working on that issue from other starting points, and Hays looks forward to collaborating with researchers in computer science, in engineering and in cognitive and linguistic sciences. He is one of the few researchers to use such an enormous visual database — those tens of millions of ordinary photographs — and he is eager to continue his memory-intensive approach in his work at Brown.

Where that might lead is anyone’s guess, but Hays expects to see dramatic progress on scene understanding in a matter of decades: “In my vision of the future, I want computer agents that are interacting with us and the world. It will be a massive economic and social change to have functional humanoids. It’s going to be one of the biggest changes of our society in the next 50 years, and I want to be part of it.”