Ethernet goes wide

By Don Ellis and Dinesh Mohan

Nortel Networks

Long known as the reliable workhorse of the enterprise, Ethernet is getting a boost from new projects underway at IEEE. As a result, the power of this simple, yet accomplished protocol will be unleashed, allowing it to become a true pillar for service providers wanting a new packet-switched infrastructure based on Ethernet.

Ethernet has essentially been transformed from a local area network (LAN) technology to a metro-area network (MAN) and wide area network (WAN) technology. Its value in enterprise infrastructures and applications networking has entrenched it as a basic connectivity layer in every size enterprise, from the extremely large to the extremely small. Its value proposition in a service provider network consists of multiple access options, scalable link rates, switching solutions, volume pricing (cost-effectiveness) and plug and play operations.

It's no surprise, then, that most service providers today have one or more Ethernet services in their portfolio. However, they have been challenged with the solution sets available to them to offer appropriate network and service operations with carrier-grade qualities. There is growing pressure to offer new, higher-bandwidth services to both residential and enterprise customers, and this in turn is driving the need to evolve from circuit-switched to packet-based infrastructures.

So yet another transformation is about to occur that will allow future Ethernet deployments to dwarf anything seen to date by service providers. Advances in network processors, physical layer (PHY) chips, and optical devices have allowed the needed policing tasks, encapsulation functions, massive bandwidth handling, and port densities, while still holding equipment costs on the traditional Ethernet cost curve. Ethernet will be regarded as a key service provider technology for:

? Customer interface

? Network demarcation

? Access technology options

? Secure service encapsulation

? Packet bridging, switching and tunneling

? Service and network operational tool set

Standards bodies such as IEEE, ITU, MEF, and IETF have been addressing these for some time. However, a watershed set of projects recently undertaken by the IEEE (who own the Ethernet definition) promises to unleash the power of this simple, yet accomplished protocol to become a true pillar for service providers wanting a new packet-switched infrastructure based on Ethernet.

Ethernet will be regarded not as a service, but as a common infrastructure for higher layer services such as the much touted triple-play of Internet access, IP video broadcast, and VoIP. This varied mix of high-bandwidth services and infrastructure positions Ethernet well as the choice infrastructure due to its cost per megabit versus other alternatives. For example, other standards bodies, such as the DSL Forum, have started migrating towards Ethernet-based DSL aggregation.

Ethernet has attained an unmatched degree of ubiquity in the enterprise, and it is also being extended for broadband home networking as well. There are numerous reasons for this, but one view is that the combination of simplicity and rigorous specification has permitted a degree of commoditization that other networking technologies can only envy.

Challenges to service providers

This has placed service providers in a difficult position as they face a technology arbitrage situation. They are on different bandwidth/process curves than their customer base. Yet, the competitive pricing of bandwidth is dropping and the aggregate pull of broadband and enterprise applications combined with technological churn is requiring carriers to deploy new infrastructure. In some cases, these solutions employ technologies that cannot keep pace with demand. At the same time, the rise of the intelligent end systems, in the form of PCs, PDAs, and cellular phones, is increasingly undermining the service provider's lock-in on value-added services beyond connectivity.

Many of the fundamental problems with Ethernet that have obstructed it from taking center stage in the service provider space are actually well understood, and they are now being addressed. This bodes well for both the future of Ethernet, and for adopting providers.

New service provider Ethernet solutions

With the IEEE 802.3ah Ethernet First Mile (EFM) activity, providers have new standards to address different media types, different reach and rate options, and improved link operations, administration and management (OA&M). When coupled with the IEEE 802.1aj (two-port MAC relay activity), the standards will address customer interface, network demarcation, and access technology options.

In addition, with the work on IEEE 802.1ah provider backbone bridges (PBB) and 802.1ag connection fault management (CFM), Ethernet will now be provided with key architecture features that separate the service layer from the network layer. These tools will permit true hierarchical scaling and full isolation of provider infrastructure from subscriber broadcast domains. This is a significant step in making Ethernet a great fit for service provider applications and satisfies concerns over secure service encapsulation; packet bridging, switching and tunneling; and a service and network operational tool set.

Figure 1 shows the PBB hierarchy with the client physical layer, the client frame layer, the provider service layer, the provider tunnel or network layer (PBB-VLAN and PBB-MAC), and the provider physical layer. The various OAM layers are shown on the right. Each one of these OAM functions is important for successful service provider deployment.

Figure 1: General client/provider Ethernet hierarchy model

New Ethernet service features

PBB (Figure 2) offers new service features that were the missing ingredients for building successful service provider infrastructure solutions based on Ethernet technologies.

Figure 2: Provider backbone bridge frame encapsulation

The PBB Service ID (SID) is part of the PBB Service TAG or I-TAG and represents the "logical service instance" at the edge of the provider service network. This is imperative, since it allows the provider to build service inventory and then associate this with customer records in a service and network management system (NMS). The SID is now at least 24 bits and large enough to track both enterprise services and consumer connectivity.

Connectivity records can be kept for each EVPL (Ethernet virtual private line) and EVPLAN (Ethernet virtual private LAN) service. These service records can be used for enterprise retail and wholesale services, for wireless backhaul services, for consumer backhaul interconnect and general network connectivity. This represents a cost-effective way to track both service and network connectivity over Ethernet and can replace the circuit connectivity and circuit records required to track current digital interconnects.

Rate of Service (ROS) can also be offered with new rate limiting, policy, queuing, and shaping features offered via Ethernet. Each service can have bandwidth or ROS features (such as service management information bases [MIBs] and metrics) associated with the SID at the edge PBB network elements, and specific service reports can now be generated at the NMS.

PBB packet tunnel structure also enables providers to offer distance-based service pricing by using the PBB MAC address and network TAG structure, including PBB VLAN ID. This is a key part of any service provider infrastructure, yet it was ignored in early Ethernet deployments. The "death of distance" concept associated with voice and VoIP solutions may have merit, but this should not remove the value of connectivity over distance. Many other industries value distance and new technologies in each industry may improve the cost/vale ratio, but distance still has significant value in many segments (such as in air, rail, parcel, and subway transportation). Technologies that cannot easily offer distance options and features may de-value the service provider infrastructure and not enable successful business models. An enterprise that needs 10M to 1000M for server backup understands the value of distance and is willing to pay for different distance options.

Table 1 shows a typical example of circuit pricing in Asia. Although these prices are from 2003, they demonstrate a definite value in distance pricing.

Table 1: Distance and rate pricing example in annual US dollars.

A typical European or US example of circuit pricing for private line service would include installation charges, monthly rate charge, and a per kilometer charge, with different rates per contract term. The rate charge per mile for 100Mb/s could be higher than $100 month/mile. The example demonstrates that both rate and distance are key factors to be considered when planning basic infrastructure solutions.

New Ethernet networking features

The 802.1ah standard intends to address a missing ingredient from what is currently available, including Ethernet 802.3/802.1 and the follow-on Provider Bridge (802.1ad). Previous approaches left limited address space for service and network features. The PBB adds a complete provider MAC address and network TAG, including the PBB-VLAN (which allows the provider to construct separate provider domains).This PBB-VLAN can be implemented to construct zones which the provider can administer for the various service types. The PBB-VLAN can represent the logical tunnel structure across the network.

This is important because it allows the provider to build tunnel inventory in the PBB network elements which can also be associated with network records at the NMS level. The PBB-VLAN field is 12 bits and is large enough to associate VLANs with both distance and for network topology (pt-pt, tree, bus). The options for pt-pt topologies may be useful for some enterprise service offers, while tree topologies may be useful for consumer video distribution (IPTV). The bus topology may be suitable for best-effort internet access service applications.

Such a zone of service (ZOS) concept is an example of how PBB networks could be used to replace digital circuit infrastructure and still try to preserve a network distance structure to track cost and pricing options.

Table 2: Distances and zone pricing example: monthly price per Mb/s (in US$)

Tables 2 and 3 show how distance pricing could be offered and each service can be offered across different zones at different pricing levels. The price per Mb/s per month is shown in Table 2. The full rate port pricing per month Table 3 for a 100Mb/s EVPL service at the metro zone would be $2K + $20x100Mb/s = $4K per month.

Table 3: Distances and zone pricing example: monthly full rates (in US$)

New scale features via hierarchy

Due to its nature, Ethernet has faced a number of scaling challenges. The combination of a flat address plan in the form of MAC addresses and the use of broadcast and multicast auto discovery mechanisms has typically placed limits on the upper bounds of sizing an Ethernet network. Previous attempts to address this have been to partition or segment the network. This was achieved with the use of VLANs, a technique primarily of utility to the enterprise. Extensions to VLAN tagging specified under the Provider Bridge effort went further in allowing providers to impose their own partitioning without disturbing customer partitioning through stacking of the VLAN tags.

The use of hierarchy is well understood for scalability and security. Hierarchy is particularly useful when the subscriber base is a large number of relatively small communities of interest that can be overlaid upon a common transport network. It reduces the amount of provisioning and forwarding state in the network core (impacting OPEX and CAPEX costs) and correspondingly reduces the load of performing service assurance and fault management.

Hierarchy has a number of security implications as well. Subscriber isolation and containment means significantly less complexity is required at the edge to police customer behavior. It can also be said that auto-discovery (one of Ethernet's strengths) and security are polar opposites, yet, much of Ethernet's plug-and-play capability is highly desirable in terms of operational complexity. Hierarchy puts the plug-and-play back into carrier operations. The transport layer interconnecting the carrier edge devices can operate in a plug-and-play mode as the network edge is secured physically. The network edge encapsulates and constrains subscriber traffic received from outside the secure boundary.

New Ethernet OAM improvements

Significant progress has been made in the field of data plane OAM with efforts in both the ITU-T (Y.17ethoam) and the IEEE (802.1ag). There is strong agreement as to the functionality required, and a comprehensive suite of tools is already emerging. When the data plane forwarding and relay behavior is clearly understood and well specified, procedures for verification become a similarly well understood problem. The end result will be an increasing degree of front line reliability and much lower mean-time-to-repair when failures happen. It will also increasingly decouple the need for the Ethernet network to depend on transport OAM (e.g. SONET/SDH) or client OAM (e.g. ICMP Ping), neither of which is truly native to the maintenance entity of real interest. In the long term, it will permit additional resiliency options to be added and for comprehensive instrumentation of service level agreements (SLAs).

Both IEEE 802.1ag and Y.17ethoam have started addressing some network management fault, configuration, accounting, performance, and security (FCAPS) features. While IEEE 802.1ag is addressing OAM improvements for fault management, ITU-T Y.17ethoam is addressing both fault management and performance management. The OAM associated with the provider service ID will offer additional service management features and will result in improved fulfillment, assurance and billing features. Other important carrier requirements are also being addressed. For example, IEEE 802.1AB defines data plane procedures and management interfaces for topology discovery, a key pre-requisite to NMS and OSS integration.

When considering Ethernet as a potential convergence technology, standards are emerging across the spectrum of Standards Organizations. The Metro Ethernet Forum has issued a number of recommendations related to Ethernet services, both in terms of Ethernet presentation to the subscriber, but also other presentations such as adapting circuit emulation onto Ethernet. The ITU has considered a number of recommendations defining carrier Ethernet services (Ethernet private line, and virtual private line).

Table 4: A summary of the IEEE, MEF and ITU Ethernet related activities.

The DSL Forum's architecture and transport working group has been busy addressing the migration of carrier metro infrastructure for broadband from legacy ATM to Ethernet. There are many drivers for this, including the need to drive significantly higher bandwidth cost effectively. Working Text 101 "Migrating to Ethernet Based DSL aggregation" is intended to be a comprehensive view on use of Metro Ethernet networks to provide multicast capability in addition to interconnecting DSLAMs and access nodes to broadband remote access servers (BRASes). It is expected that PBB will also add value in the progress of this work as well.

The bottom line

For service providers, the end result of the current round of standardization is that Ethernet is ready to step into the limelight, not just as a link technology, not just as a customer presentation, not as OAM deficient technology needing assistance from its neighboring layers, but as a full-fledged, complete, cost-effective and "fit-for-purpose" infrastructure layer in its own right.

Ethernet activities are being developed to deliver tightly specified quality of service (QoS) levels on a truly massive scale, with predictable performance to a varied customer base. Challenges will remain, but the industry looks poised to deal with them as they arise.

About the authors

Don Ellis is senior advisor in the Nortel Networks Optical/Wireline Product Strategy group. Dinesh Mohan is a senior standards advisor in the Chief Technology Office at Nortel.