Global Sources
EE Times-Asia
Stay in touch with EE Times Asia
EE Times-Asia > Embedded

Parallel processing architecture enables advanced vision apps in today's cars

Posted: 02 Apr 2007 ?? ?Print Version ?Bookmark and Share

Keywords:automotive safety? inside vehicle protection? equipment accident avoidance?

- By Jens Eltze
NEC Electronics America

Safety continues to be one of the key concerns in automotive design. In addition to keeping drivers and passengers safe, automotive manufacturers are looking for ways to protect people outside the carsuch as bicyclists and pedestrians. While airbags and seat belts are designed to provide protection inside the vehicle, accident avoidance equipment, through applications such as vision-based warning systems, could provide the ultimate solution for protecting lives both inside and out.

Analysis shows that the vast majority of accidents are caused by human error or misjudgment. Automotive electronics that could monitor driving conditions and warn a driver at the onset of a hazardous situation could go a long way in preventing and mitigating human-induced accidents.

When vision-based warning systems for drivers were first introduced, acceptance from consumers was limited, mainly due to high costs and a lack of reliability related to false warnings and such. Systems have improved dramatically in recent years, and new vision systems are now being introduced by companies such as Toyota for its flagship Lexus LS460 (below). Current technologies enable automotive designers to implement robust and powerful recognition algorithms while meeting the stringent power consumption and operation constraints associated with automotive design.

Toyota's flagship Lexus LS460

The challenges: performance and power consumption
One key consideration when developing a vision system for cars is the need for rapid processing of camera images. For example, a car moving at 75mph would travel about 110ft/sec. Common video cameras capture images at a rate of 30fps, which means the car would travel 3.6ft between the time an image was captured and the time it became available. To recognize hazards within 7ft at this speed, a vision system would have to be able to analyze images before the next image was captured.

Such performance does not come easily. In the Lexus LS460, the system analyzes images of lane markers, nearby objects and pedestrians that are captured by two stereo cameras.

In an automotive system using a conventional microprocessor or signal processor, the dedicated hardware filters preprocess the incoming image before the main processor can do its analysis, as the microprocessor or signal processor does not deliver the required performance (Figure 1, left).

Figure 1: In a conventional system (left), time-consuming preprocessing needs to be done and hardware-based logic hampers changes or upgrades. A software-based, parallel processing architecture, such as IMAPCAR, facilitates processing speed and changes (right).

In addition to requiring dedicated hardware logic for filtering, a conventional system also limits the possibility of adding simple algorithm upgrades if a change in the hardware filter is necessary, or of implementing new features if a system's pre-filters conflict with the filters for the existing features.

NEC Electronics developed a solution, now called IMAPCAR, that enables system developers to implement all filters and recognition algorithms in a high-level software language, eliminating the need for hardware filtering (Figure 1, right). The key challenges included not only achieving performance high enough to do even complex filtering in software, but also keeping power consumption for the device below 2W. Any power consumption above 2W usually requires dedicated heat-loss countermeasures when operating under the automotive temperature range (-40C to 85C). This impacts not only system costs, but also limits the options for mounting the system in a car.

To overcome these challenges, members of NEC Electronics Automotive Systems Division teamed up with engineers from the company's in-house Multimedia Research Institute (MRI). Using experience in the field of real-time image processing, MRI designers developed an architecture based on extensive parallelization. The Integrated Memory Array Processor for Car Electronics (IMAP-CE), unveiled at the 2003 International Solid-State Circuits Conference (ISSCC), delivered peak performance of 50GOPS.

To put this number in perspective, the image processing capabilities of IMAP-CE were benchmarked in 2003 against a leading 2.8GHz processor used for the PC market by running various standard image filters. IMAP-CE performed 3 to 30 times better than the PC filters hand-coded in assembly and 10 to 50 times faster than the PC filters written in C language.

Still, that performance was only about half of what was needed for a robust implementation of the image recognition system used in the LS460. And the team also needed to address the power consumption challenge, as implementing dedicated heat-loss countermeasures is not really an option in automotive applications.

To keep power consumption low, the team set the device's target operating frequency to 100MHz and based the overall design on one of NEC Electronics' small-geometry processes for automotive devices. The result: the team was able to successfully enhance the device's internal architecture, enabling the definitive IMAPCAR device used in the Lexus LS460 to deliver 100GOPS of peak performance and achieve power consumption as low as 1.7W.

IMAPCAR architecture
The IMAPCAR processor is based on a single-instruction multiple-data (SIMD) concept. The SIMD array consists of 128 processing elements that can operate in parallel. The internal architecture has been optimized for image processing, and each element can execute up to four instructions at the same time. The interconnectivity between the processing elements allows efficient exchange of information (Figure 2).

Figure 2: The IMAPCAR architecture features 128 elements that parallel process data from up to three cameras.

The control processor manages the operation of the device, controlling the flow of instructions with an internal pipeline, broadcasting the instructions to the SIMD array and handling the memory interfaces.

One unique feature of the IMAPCAR processor is its ability to handle inputs from up to three cameras in parallel, making it suitable not only for single but also for stereo camera applications. An external host controller can communicate with the IMAPCAR processor through its dedicated system control interface, picking up the processing results and handling the integration in a car network.

Software implementation
The IMAPCAR processor implements all filters and algorithms in software. Therefore, it was important for the design team to support high-level-language programming, as assembly-coded routines are neither very efficient for the software programmer nor easy to handle for code verification and validation.

To help software developers employ the SIMD architecture most efficiently, the team implemented a dedicated extension to the C language (1DC) for full support of SIMD data types, operators, and syntax. Reference libraries stocked with the most commonly used filters provide a quick start for new software developers. A debugger, simulator and other development tools offer full support at any stage for the verification and testing of algorithms and routines.

Systems based on the IMAPCAR processor can be updated or upgraded at any time through a simple software change. All filters are realized in software, eliminating hardware pre-filters and avoiding the concerns over filter conflicts. This capability ensures that new algorithms, and even new applications, can be implemented smoothly.

The road ahead
Vision systems are commonly considered to be one of the key elements for the transition from passive safety systems to active safety systems. These systems can detect hazardous situations at an early stage, and take preventative measures to both reduce the likelihood of a crash and protect the occupants if a collision does occur.

The list of applications that can use image recognition is long (Figure 3). In the interior, cameras can monitor the position of the driver and the passengers in real time, enabling the personalization of interior safety systems tailored to the seat occupants' physiology and position. Systems can also warn the driver if he or she gets distracted. Outside-viewing cameras can be used to cover blind spots, view the road ahead, increase vision at night, read traffic signs, and check traffic lights.

Figure 3: Automotive vision applications range from monitoring passengers and drivers within the vehicle to providing situational awareness in the space surrounding the car.

Flexible vision system architectures, such as the NEC Electronics IMAPCAR processor, provide automakers with new approaches to road safety that will support the number and variety of vision applications that develop in the future.

About the author
Jens Eltze
is the principal technical application engineer, systems engineering, for NEC Electronics America's automotive strategic business unit. Eltze received a master's degree in electrical engineering from the University of Karlsruhe, Germany.

Article Comments - Parallel processing architecture ena...
*? You can enter [0] more charecters.
*Verify code:


Visit Asia Webinars to learn about the latest in technology and get practical design tips.

Back to Top