The Network Backbone - Not Just an IP Highway

Posted: 03/1999

The Network Backbone – Not Just an IP Highway
By Charlotte Wolter

Internet protocol (IP)
architecture in the backbones of large long distance carriers is a new phenomenon in
telecom. As recently as two years ago, most network providers had no IP in their backbone,
except as traffic carried along on another transport technology.

Today, big carriers are putting raw IP directly onto their optical networks as IP over
synchronous optical network (SONET), IP over dense wave-division multiplexing (DWDM) or
even packet over optical. IP has been tapped as the "converged" network
technology of tomorrow: a single economical network architecture that will carry any kind
of traffic and provide any level of quality of service (QoS) the traffic demands.

But is IP up to the task? Clearly, it has gained increasing favor for a number of
applications. Internet service providers (ISPs) and the network providers who service them
are using more and more pure IP backbone.

It also has spawned one of the more acrimonious debates among telecom technologists, as
IP proponents have squared off against those backing another newer technology,
asynchronous transfer mode (ATM), as the king of the backbone services.

In 1998, IP was in the media spotlight, but ATM led the way in new users. "The
story that grabbed the headlines in the trade press was the ascendance of IP, and ATM is
supposedly yesterday’s news and a corpse by road," says John Ryan, chief analyst,
Ryan Hankin Kent Inc. (RHK), San Francisco. "In fact it was a terrific year for ATM
and will be again this year."

In the real world where network providers must serve customers and make money, both
technologies are deployed–IP not as widely as ATM, but growing. And they have to share
the spotlight with other legacy technologies, notably frame relay, that have provided very
effective solutions to particular customer needs.

For the service providers, the real-world demands of customers have dictated that they
come up with their own solutions to the question of which will be the converged network of
tomorrow. For carriers, the overwhelming requirement has been that they must be able to
carry a great variety of traffic, from time division multiplexing (TDM) voice to IP. No
matter what their predominant backbone strategy, they have to make accommodation for this
diversity in their payloads.

In the ATM-dominant networks, the strategy is to encapsulate everything in ATM packets
and simply compensate for the increased overhead with more bandwidth. ATM has the ability
to carry almost any traffic, but the packet structure adds about 10 percent in overhead.

Another strategy has been to build several parallel networks. A wide range of operators
has used this. IXC Communications Inc., Austin, Texas, added its new all-IP
"hierarchical-meshed" network beside an existing SONET-based network. Other
operators, such as AT&T Corp., have IP, ATM and frame relay running simultaneously.

QoS demands expected from customers also is a critical factor in choice of technology.
Where there is expected to be a demand for mixtures of voice, video and data–as with
Sprint Corp.–ATM is a technology of choice. As in networks designed to service ISPs,
where data represents the dominant feature and voice is an add-on, IP is the dominant
technology.

Private networks for intranets or virtual private networks (VPNs) are predominantly
IP-based now be-cause they carry mostly data. However, as corporations gradually integrate
voice with data in their enterprise networks, they also will be carrying voice over IP
(VoIP).

This does not necessarily mean that these networks will migrate to ATM for QoS. Being
private, they can tightly control use of the bandwidth to keep the quality up for VoIP.
Therefore, it is considered likely that they will remain IP-based, even as voice becomes a
larger part of their traffic mix.

One overall trend that will favor IP-based networks, experts say, is the burgeoning
growth of data on all networks. By 2005, according to several estimates, data will
outweigh voice by a factor of 30 to one on backbone networks, a powerful force that is
expected to bring IP into predominance.

Sprint on Cruise Control with ATM

As network operators have made their choices, some interesting solutions to the
question have emerged. Sprint has been one of the most vocal proponents of all-ATM
backbones, and is well along in its deployment of services. "We have technologies
that we are in the process of deploying–one in particular that runs voice over ATM that
allows us to talk to the PSTN (public switched telephone network)," says Charles
Fleckenstein, technology spokesperson. "IP at this point, instead of being a
transport mechanism, is still seen as an application. We don’t see IP on the backbone, but
consumers would see IP in the home."

Sprint’s arguments for ATM are flexibility and quality. "What we can do with ATM
is, it can roll up IP, frame relay, voice and video, and run them all over a single ATM
backbone," Fleckenstein says. "ATM is the only technology that gives the quality
of service we are looking for."

One of the efficiencies that Sprint hopes to introduce in its long-term vision of its
nine-month-old Integrated On-Demand Network (ION), is an element of subscriber management
and provisioning of their own services, even for consumers. Each subscriber premises, home
or business, is equipped with a hub that talks to a service node on the edge of the
network. Services are provisioned on the service node. A customer chooses a service, such
as conference call, and the hub talks to the service node, which automatically dedicates
bandwidth to the user. If it is a voice call with highest QoS requirements, the network
will automatically allocate 64 kilobits per second (kbps); if it is asymmetrical digital
subscriber line (ADSL), as much as 6 megabits per second (mbps) will be dedicated to the
service. "With ION the idea was customers are never turned off, like
electricity," Fleckenstein says.

Ryan says it is true that Sprint is making great efforts to use ATM "as a sort of
convergence technology so they can carry voice and frame relay and ATM and X.25 in a
single managed infrastructure. That is a lot of what they are trying to do with the ION
network."

Beth Gage, a Tulsa, Okla.-based analyst with TeleChoice Inc., Owasso, Okla., says,
"For IP-type services it would be an advantage in not losing overhead, not adding
latency in processing times. You do add latency when translating IP to ATM and frame relay
back to IP. If you are in a pure IP environment, then you don’t lose anything to that
translation."

However, at the company’s own ISP, the Sprint Link Group, the backbone technology is
packet over SONET. "So, if you look at their network, they have IP routers, and the
output goes directly into a SONET terminal without passing through a Layer 2 box–ATM or
frame relay," Ryan says.

"So what does that prove? What it proves is that at this particular juncture the
attempt to create a single converged network is extremely premature. Instead, it makes
senses to separate the needs of pure IP networks from other networks."

Qwest Rides IP

In apparent contrast to Sprint–but very much like the architecture of the Sprint
ISP–is Qwest Communications International Inc., Denver, Colo., which is building a
nationwide backbone of IP over SONET with OC-48 capacity. About half of the company’s
intended mileage of 18,500 miles is "lit," with the rest planned to be in
operation by June 1999. A number of customers are connected already, including Netscape
Communica-tions Corp., Mountain View, Calif., which has OC-12 capacity. Qwest recently
announced it had reached 1.5 million miles of OC-3 (155mbps) capacity. The network also
includes some capacity of OC-192 using IP over DWDM.

The company says the network has one-to-one redundancy, with
50-millisecond-to-200-millisecond restore times and extended switching down to the router,
so customers are not aware of downtimes.

"Due to these capabilities, we are able to provide the same quality of service and
uptime that customers are used to in the old telephony world," says Vab Goel, Qwest’s
director of IP network engineering, "but at a cheaper cost with more flexibility for
leased types of capacity and phone."

The services include IP data, VoIP (launched late in 1998) multicast-enabled video for
videoconferencing, and the network can carry frame relay and ATM for private-line
services. "We have thrown a lot of capacity on the backbone. We did not build it just
for web traffic," Goel says. "We are not offering it like an ISP. It is not a
backbone; it is an infrastructure."

Goel says the trend in private networks is that companies are going to build them over
public networks using VPN technology. "VPN is going to be the key feature set offered
on our backbone, and there is a tremendous amount of demand in the marketplace for
companies to move out of their legacy frame relay or ATM networks and get onto VPN so they
can take advantage of the flexibility of the IP protocol," he explains.

Qwest has parallel nationwide frame relay and ATM networks for those customers still
concerned with getting value out of their legacy equipment. Its parallel networks offer
customers a migration path that will enable them to get from frame and ATM to IP.

Goel says QoS is not an issue on the Qwest network, which is built for 0 percent packet
loss and real-time operation. The capacity in the backbone provides enough bandwidth to
ensure QoS. Qwest also achieves QoS with one-to-one operation. "Other IP backbones do
oversubscription of five-to-one or 10-to-one," Goel says. "We have the capacity
to do one-to-one, so we don’t drop packets and, in case of failure, we have SONET
autoswitching. So we are able to provide real-time applications on the network."

Qwest, with its brand new, built-to-order IP network, is fairly unique in its ability
to call up more bandwidth when it has to ensure QoS. "Not everyone is in Qwest’s
situation, so it’s not an answer anyone can use," Gage says, "and networks tend
to fill up pretty fast anyway."

Tacking on more DWDM channels is not necessarily the answer, she adds. "Even with
new carriers, like Williams (Tulsa, Okla.) and Qwest, and technology like DWDM, [try to]
balance that with the phenomenal growth of the Internet and the continuing evolution of
data applications," she says.

With cable modem access and DSL just starting to hit the market, bandwidth usage will
ramp up considerably over the next five years to 10 years. "What happens to all this
space when we have 10 million users with 1mbps access to the Internet?" Gage says.

"Fundamentally it is an unresolved issue," Ryan says. "The
terabit-router people are not the only ones who think overprovisioning the network with
more capacity than needed makes sense. The unresolved part is that it’s not clear whether,
in fact, it is cheaper to build a massive network than a very good network, and no one
knows the answer to that one."

AT&T Changes Lanes

While Sprint and Qwest espouse a particular network architecture, AT&T is carrying
the torch for technology agnosticism. The company’s architecture carries a mix of backbone
technologies, each aimed at a different service category. The basic network is SONET or
DWDM, with private-line, frame relay, ATM and IP transports all sharing that platform.
Until recently, the whole network was SONET, but recent developments, including a deal
with @Home Networks, Redwood City, Calif., for OC-48 DWDM channels, has put some IP
directly on to DWDM.

The network has 35,000 miles of SONET in 57 rings across the United States. The choice
of backbone technology comes down to customer needs rather than absolute considerations
about QoS, says Dan Sheinbein, vice president, network architecture and development for
AT&T. Putting IP directly onto DWDM means the network loses some of the protections of
SONET. If there is a fiber cut, the network may not be able to achieve the quick
100-millisecond restore time of SONET. And, because an IP network is fully meshed, a cut
may mean that 2.5 gigabits per second (gbps) of traffic suddenly gets dumped on other
routes that may or may not have the capacity to accommodate them.

"Many ISPs are saying, ‘I don’t want you to go through the SONET layer and give
100-millisecond restoration. I will do it internally to my network. Just give me OC-48
straight into DWDM,’" Sheinbein says. "We at AT&T are agnostic. If the
customer does want fast restoration, and most non-ISPs do want it, we will provide
it."

In its own ISP, AT&T still is using the SONET layer, "but we will see how
things go," he adds.

The actual traffic carried by the different transport methods in use in the AT&T
network–ATM, frame relay, IP–is usually IP, Sheinbein says. But the network continues to
use frame relay and ATM because of their ability to deliver guaranteed bandwidth.
"You can’t do that with IP today, and I emphasize today. IP is a send-and-pray method
of transmission," Sheinbein says, "But IP will get there, and the reason is
because AT&T and other carriers will begin to provide virtual private networks using
IP. As we get more security and screening functions and can guarantee capacity in IP, we
will see more and more customers use IP."

Sheinbein also believes frame relay and ATM will not disappear, because of large
installed bases and the difficulties of migrating from one network technology to another.
"So we are more likely to see interworking between frame and ATM networks, which we
have begun, and between those networks and the IP network as we get more QoS [in IP]. As
we move forward, we will see lots more IP on the backbone," he says.

For VoIP, AT&T provides its own consumer VoIP, called Connect and Save, that uses
two-stage dialing and essentially is taking advantage of IP arbitrage opportunities.
AT&T uses overengineering to ensure QoS on that service.

A newer play, but the one that is expected to grow exponentially, is VoIP over
intranets. Here, some companies are using overengineering for QoS while others are doing
the opposite, compressing the voice and putting it in gaps in the data transmissions.
Today, this is used primarily for communications within a company, particularly between
remote offices, with public branch exchanges (PBXs) directing some traffic onto IP and
other traffic onto the switched network.

But, compared to data, IP still is rather thinly deployed in the voice realm, and one
reason is that backbone providers are being fed so meagerly with IP traffic by incumbent
local exchange carriers (ILECs) and competitive local exchange carriers (CLECs). That is
not surprising, considering that VoIP to consumers is seen as a very long-term play in the
incumbents’ local loop. CLECs are expected to see more IP voice, particularly as corporate
customers start to integrate voice and data networks, but that market has been slow to
develop.

"Most CLEC applications that we have seen so far for IP have been rather
limited," says Cody Bowman, AIN product marketing manager, Ericsson, Stockholm
Sweden. "We talked to a number of customers over the past six months to a year about
where they see IP unfolding in the local loop, but overall we see VoIP applications tied
more to interexchange carrier (IXC) use."

IXC Builds New Road

IXC Communications has taken pure IP to a new network architecture that combines some
features of SONET rings with IP’s "meshed" architecture, one it calls
"hierarchical meshed." Basically all routers in the network eventually are
connected to a mesh as in a typical IP network, but they are organized into eight
"regions" that act as sort of hubs or "superrouters." They also enable
cross-country links of just two hops, from one regional hub to another, which is virtually
unknown in other IP network topologies.

IXC Communications puts packet over SONET, with very-high-capacity DWDM-based links,
now at OC-48 and able to scale to OC-192. IXC has revealed few details about pricing and
specific services available on the network, leaving analysts unsure about its impact as a
new approach to IP. "A lot of companies are talking about all the bandwidth they can
deliver and high-speed backbones, some packet over SONET, packet over optics, ATM over
SONET," says RHK analyst Joe Struber. "To me, the real issue is what are the
services they can deliver and how are they priced? Maybe it’s a great packet-over-optic
network, but what if it costs twice as much as packet over ATM or I can get it only in
three cities?"

The biggest problem for IP at this point is likely a QoS issue, which is far from any
resolution. "If you get into multimedia traffic over IP or voice over IP added to
data or if you want to have different quality of service for different applications or
different customer classes, that is still where IP has some maturing to do," Gage
says, "and new protocols, such as DiffServ (differentiated services) and MPLS
(multiprotocol label switching), are being defined to address that."

A huge part of the problem, Ryan says, is that currently there is no way to control
routers in IP networks so they either set up a virtual circuit or look farther ahead when
choosing routes. If network operators could just tell a router which route to use,
"the estimates that we have seen are that traffic engineering like that can save 25
percent to 30 percent of capital cost, not just bandwidth," Ryan says. The
alternative way to solve the problem of a congested link would be to expand the router and
buy more bandwidth, more OC-48s. And capacity on other routes in the network could
continue to be underutilized, a stranded investment. "So the debate over how to build
next-generation, high-capacity networks remains a very, very vigorous debate," Ryan
says.

Carriers Drive Defensively

In sum, despite bold pronunciations by some of the newer network operators about the
ascendancy of IP, none are running truly IP-only networks. And, even an ATM stalwart such
as Sprint operates a pure IP-over-SONET architecture for its ISP.

"The answer is, because someone has to pay the rent," Ryan says.
"Although for the services that are most mature, meaning technologically most stable,
there is more money in them, they are not native IP services. They are voice and data,
like frame relay, maybe IP inside the frame relay, but the company is signed up for frame
relay service or signed up for ATM."

RHK estimates that IP revenues are well below 10 percent of telecom services revenues.
"Here is where the big argument for creating a converged network becomes more
difficult," Ryan says. Nevertheless, he says, reliable industry estimates are that IP
traffic is rising each year by a factor of 10. If that estimate is true, within a few
years voice will be a very small part of an IP-dominated network. That network may well be
close to the 30-to-1 estimate of the ratio of IP data to voice traffic.

The fly in the ointment is that IP service revenues, however, are not also increasing
by a factor of 10 or eight, but in fact just doubling. "So there are interesting
strains on network architecture that don’t, in the short term, point to happiness in
converged networks," Ryan says.

Charlotte Wolter is infrastructure editor for PHONE+ magazine.