Breakthroughs in computer vision seen by DARPA

"I have quite a bit of hope to do much more intelligent things that compare machine capabilities to those of humans," stated Donlon, who manages the Mind's Eye computer vision program at DARPA.

The Mind's Eye program aims to develop breakthrough algorithms for automatically recognizing and describing human activities. Donlon showed small steps forward—and a few bloopers—from his first 18 months of work on the three-year effort.

For example, efforts of a dozen systems failed to recognize a running dog. One described a collision between two shopping carts as "the car left." In particular, current algorithms have difficulty detecting forearm motions that are key to activities of high interest such as giving and taking.

"Classification approaches break down horribly when you get to anything transactional," said Donlon. The Mind's Eye program has developed a data set of 7,676 real-life videos it has contributed to the computer vision community in an effort to challenge algorithm developers. Researchers tend to focus on major advances with existing, sometimes artificial and academic data sets.

"The Mind's Eye data set hopefully will provide the next level of challenge the vision community needs," Donlon added.

Donlon, manager of the Mind's Eye computer vision program at DARPA.

"The CV community is more wired to making incremental bits of progress on fixed data sets than to reward less-good results on messy [but more realistic] data sets," he said. "Mind's Eye can coax people out of defined data sets. My role is to create those conditions so you can make those breakthroughs happen," he continued, noting he deliberately put "confounding factors such as variable lighting" into his videos.

"You can't underestimate the importance of the algorithms which have already come a long way," said Bruce Kleinman, VP of platform marketing of Xilinx.

Although the program's focus is on enabling breakthrough algorithms, it includes system integrators providing implementations of new code in FPGAs, GPUs and SoCs. "The hardware acceleration has greatly heightened the research," he said.

"The U.S. Army has the very sensible goal of taking a robot with a camera and have it navigate to designated points, erect a camera and stream video back," replacing a human scout, Donlon said. "Clearly what we need to do is put the intelligence on board, on the edge of the sensor."

"We need to recognize activities currently out of reach of the technology, the verbs, the action and the narrative of the scene," expressed Donlon who has identified 48 target actions to recognize. "Then we need to do operationally relevant things such as describing scenes and filling in gaps including attempts of people to fool the camera."

- Rick Merritt
??EE Times