Global Sources
EE Times-Asia
Stay in touch with EE Times Asia
EE Times-Asia > Power/Alternative Energy
Power/Alternative Energy??

Paving the way for Exascale computing

Posted: 30 Mar 2015 ?? ?Print Version ?Bookmark and Share

Keywords:power management? Exascale computing? supercomputer? Blue Gene/Q? BGQ?

Here's a little a pop quiz. What do Finland, seawater and the path to Exascale computing have in common? Hint: it's not rollmop herring. That only accounts for the first two.

The answer is power management, and specifically the problems encountered when you try to shoehorn a large amount of computing power into a constrained space, no matter whether it's a data centre or a supercomputer.

Finland is where Google located their Hamina data centre. They picked that country for two main reasons, both related to power management: cheap electricity and a cold climate. The seawater is used to cool the data centre C it's located in a former paper mill which has ? mile seawater tunnels which are just the ticket for piping in coolant to the data centre heat transfer unitsstraight from the Bay of Finland.

And Exascale computing? It's the next big goal in supercomputer performance C and power management is one of the main obstacles to getting there.

How fast will an Exaflop machine be? To give you an idea, imagine performing one calculation per second starting at the Big Bang and continuing to the present day. Now compress that into 1 second. You'd be slightly less than halfway there.

Current supercomputers achieve their petaflop-level performance through massively parallel architectures which use megawatts of power C take a look at the current top performers as of November 2014.

Table 1: The TOP500 (source:

Interestingly, the number 2 and number 6 systems use NVIDIA graphics processing units (GPUs) originally developed for video gaming, to achieve that performance. In fact, 75 of the top 100 systems use GPUs from either NVIDIA, ATI Radeon, or Intel (Xeon Phi).

The number 1 machine, the Tianhe-2 located at Sun Yat-Sen University, makes use of 32,000 Xeon Ivy Bridge processors together with 48,000 Xeon Phi co-processors, for a total of 3,120,000 cores.

And it consumes an eye-popping 17.8MW of power, enough to supply some 5100 homes, which would run around $18M a year with US electricity rates. And that's not even including the cooling system, which itself uses another 6MW.

Performance-wise, the Tianhe-2 clocks in at around 33 Petaflops (33 x 1015 floating point operations per second). It's pretty impressive, but that performance is only 1/30 of the exaflop level:1018 operations/sec.

Current-generation power management
For an in-depth look at a current-generation power management system, let's examine IBM's Blue Gene family of machines. The latest generation, Blue Gene/Q (BGQ) is used in 24 TOP500 systems, including Sequoia (#3) and Mira (#5).

For this, EDN had the opportunity to sit down with Dr. Paul Coteus, IBM Fellow and one of the chief engineers behind the Blue Gene family power system, and we are indebted to him for his insights.

BGQ hardware overview
The hierarchy of Blue Gene modules is shown below. A rack of a BGQ system consists of two midplanes, eight link cards, and two service cards. A midplane contains 16 node boards. Each node board holds 32 compute cards, for a total of 1,024 nodes per rack.

Each compute card has a single 18-core PowerPC A2 processor16 cores for applications, one core for system software, and one core inactivewith four hardware threads per core, with DDR3 memory. BGQ thus has16,384 cores per rack.

Figure 1: Blue Gene/Q Architecture (Source: IBM)

Power supply overview
How does IBM go about supplying almost 8MW to 1.6M microprocessor cores?

A block diagram of a BGQ power distribution system is shown below. This one is from Mira, a 48-rack BGQ system installed at the Argonne National Laboratory, #5 on the TOP500 list.

Figure 2: MIRA power system (Source: Measuring Power Consumption on IBM Blue Gene/Q)

1???2???3?Next Page?Last Page

Article Comments - Paving the way for Exascale computin...
*? You can enter [0] more charecters.
*Verify code:


Visit Asia Webinars to learn about the latest in technology and get practical design tips.

Back to Top