Ringed bus technology puts twist on switching networks

Switch-fabric technology has existed for many years in mainstream applications embedded at the core of the network, such as ATM and Ethernet, in the form of proprietary ASICs designed by networking equipment vendors. But the increasing reliance on networking and internetworking access equipment, and the need for "five-nines" (99.999 percent) reliability, are placing demands for availability, reliability and scalability (of bandwidth, slot/port count and distance between chassis) on bus-based interconnection at the edge of the network, both inside and outside the box. In particular, networking equipment designers have begun building switch-fabric-based systems aimed at service providers, which are experiencing this pressure most intensely.

The shared-bus architectures used in communications equipment are based on proven standards-based I/O buses, and will continue to be successful in line cards. PCI, for example, remains the architecture of choice for communications equipment because it is based on industry standards, is extremely cost-effective, has a wide range of development tools, and has an established engineering expertise. However, these buses must now be complemented by a revolutionary backplane architecture that can address existing and future needs at the network edge.

As a consequence of all this, new PCI-based switch fabrics are emerging. They are embedded in networking systems such as DSL access multiplexers, WAN access equipment, cellular base stations and mobile switching centers, and remote access gateways. Some switch fabrics provide scalability and reliability to existing backplanes, while also preserving investments in PCI architecture.

However, designers of this first-generation network edge/access equipment using switch fabrics have spent a disproportionate amount of time and money designing switch fabrics into their systems. They have done this because there are no standard switch-fabric architectures designed specifically for their applications. System designers need an architecture that not only breaks the interconnect bottleneck of buses, but also preserves their investments in PCI; meets their varied performance, cost, availability and reliability targets; is highly scalable; and is practical to build and manufacture.

The term "switch fabric" refers to several different, but related, levels. These include an interconnected network of many multiport switches, or switching devices, used throughout a large geographic area. Together, they comprise a routing infrastructure; the internal interconnect architecture used by a switching device, each of which redirects data coming into one of its many input ports to one of its many output ports; and more recently, silicon switch-fabric devices, typically ASICs.

A switch fabric, in essence, is a fast, concurrent and transparent mechanism for transferring data from multiple sources to multiple destinations within a network via high-speed point-to-point links. An "any-to-any" network of high-bandwidth nodes is created, hence the "fabric" metaphor.

Multiple CPU complexes and I/O subsystems, connected and controlled by switch-fabric silicon via bus interfaces, communicate efficiently and transparently with other complexes and subsystems. Switch fabrics are inherently scalable, and CPU complexes and subsystems can be added to them during run-time as required. Logical and physical isolation make switch fabrics more reliable and secure than bus-based architectures. Unlike bridge-extended bus-based architectures, transactions in switch fabrics do not need to traverse layers of hierarchy, because of their many-to-many topologies.

Access-system vendors want to employ switch fabrics to add scalability and high availability in next-generation remote access aggregators, scalable routers, and multiple-host fail-over servers. Using switch fabrics, service providers will be able to scale their systems quickly and easily to accommodate large numbers of new subscribers.

To date, the leading switching mechanism has been the crossbar switch, used almost entirely at the network core, and consisting primarily of proprietary designs. Network core applications require switch fabrics with many very-high-speed ports supporting a limited number of physical media and logical protocol types. In contrast, network edge/access applications need many moderate-bandwidth ports, as well as flexibility for handling different types of input (such as voice, video, data, cable, and DSL) on diverse physical media and logical protocols. They are also slightly more tolerant of node-to-node latency.

This fundamental difference between core and edge/access switching requirements demands different switching solutions for each kind of platform. The high-speed serial crossbar switches used at the core provide more performance and less flexibility than are needed at the edge. They are also expensive and consume power. Other switching solutions are therefore needed in edge and access platforms.

Existing solutions have been based on proprietary switch fabrics, or shared buses. Designers of current-generation edge/access networking equipment have spent a disproportionate amount of time and money on in-house switch-fabric ASIC development because no other alternatives have been available. Often, these in-house designs consist of combined PCI-to-PCI bridges, custom logic, and internally developed software. What these designers need are high-capacity switch-fabric ASICs from merchant silicon vendors so that they can design the next-generation platforms for large systems faster and more cost-effectively.

Some PCI bridge vendors have attempted to meet these needs by extending their bridges to add switching benefits. These PCI bus "switches" can reduce latency and improve performance on the PCI bus, but they are still bus-based bridges functioning within, and constrained by, a shared-bus architecture.

The ideal PCI-based switch fabric would be designed from the beginning as a switch fabric with developers of edge/access equipment in mind, and would be able to deliver high-availability building blocks at all levels, from chips, to software, to systems. These architectures would have built-in self-healing features, error detection, and retry and redirection mechanisms; automatic, end-to-end flow control; traffic management that allocates bandwidth fairly; and multiple independent domains for multiple host systems, each with its own separate address space and each protected for fault isolation and fault containment. In addition, hot-swap and hot insertion would be supported at the board, fabric, box, and rack levels. Finally, these architectures would be flexible enough to mix different adapters, such as Compact PCI and PCI, different bus widths (32-bit and 64-bit), and different frequencies (33MHz and 66MHz PCI clocks) on the same fabric.

Switching via rings

One such switch fabric has been developed, architected from the ground up to serve in PCI-based systems. Developed by Sebring Systems, a firm acquired by PLX Technology earlier this year, this ring bus scheme is expected to emerge in product form next year. It is PCI software-transparent, requires no software to run, and appears to the host as a set of PCI-PCI bridges. Data come into the switch fabric, are converted from PCI transactions into an exchange of fabric cells, sent across the switch fabric, then converted back into PCI transfers at the target node's switch-fabric controller.

Thus, the switch fabric is transparent to the OS and the BIOS and requires no changes in the PCI line cards or controllers. Because it is based on a very high-speed, dual counter-rotating ring instead of a crossbar, the PLX switch fabric fits naturally into backplane bus-based systems (requiring changes only to the backplane) and provides fabric fault tolerance at minimum marginal cost.

This switch fabric is scalable up to 50Gbps to 100Gbps or more, and up to hundreds of PCI bus segments. It supports Compact PCI and PCI buses, as well as 32-bit and 64-bit bus widths, and 33MHz and 66MHz PCI clocks. The switch fabric can be distributed over the line cards, or built onto the backplane. Nodes can be inserted live into the fabric. Since the switch fabric uses a ring topology, its costs grow only linearly when it is scaled.

The switch fabric supports an average of eight concurrent transactions between its nodes. Each node is connected to its two neighbors by a pair of high-speed point-to-point links. Because of its dual, counter-rotating ring topology and its routing mechanisms, the farthest distance any cell must travel is only halfway around one ring. Cells automatically choose the direction with the fewest hops, unless broken links or nodes force them to take the opposite direction. Unlike bus-based architectures, the ring's transparent, self-healing architecture tolerates single faults. The switch-fabric protocol uses sequence numbers, timeout, retry and redirection to guarantee reliable, ordered delivery. It automatically detects disconnections and reconnections via continuous connectivity monitoring.

End-to-end flow control prevents buffer overflow due to congestion at a destination PCI bus segment. A configurable link-level flow control allocates bandwidth and provides fairness. Each node has a prefetch buffer and a write posting buffer. These buffers are large enough to mask fabric latency, allowing point-to-point throughput that approaches the full speed of the PCI buses.

Configuration registers allow the multiple PCI bus segments on the ring to be partitioned into multiple logical and physical domains. Each domain has its own protected address space, and only nodes within that domain can access that address space. A system is typically configured so that each host processor can transparently configure all the nodes in its domain, and those nodes can engage in peer-to-peer transfers.

An interdomain address translation mechanism is included. Those mechanisms enable fault-tolerant systems implementation with N+1/N host redundancy and software-based failover of I/O subsystems to surviving or standby host processors.

The dual, counter-rotating ring architecture is the first PCI-based switch fabric that was designed specifically to meet the needs of edge/access network equipment developers for bandwidth, cost effectiveness, availability, reliability and scalability. That approach not only relieves the interconnect bottleneck that has plagued developers, but also preserves their investments in PCI. That includes not only hardware, but driver software as well.

The ring topology is eminently flexible in design. In the future, as bandwidth needs continue to rise and interconnectivity expands, the ring will be able to accommodate new buses as on-ramps to the switch fabric. Such new buses will be able to handle even more interfaces than PCI.

? Jack Regula, Chief Scientist

Brian Huynh, Senior Product Marketing Manager

PLX Technology