Why do TCP flows, that carry the vast majority of the Internet traffic, transmit at rates they do? A user upgrades his access line to a pipe which is able to carry 16 Mbit/s downstream. However, when retrieving data  from the Internet, the user observes that the maximum throughput is not reached. What are possible reasons for this behaviour? How can root causes for low throughput be identified? Knowledge of factors that determine TCP’s throughput is therefore valuable for users as well as network operators.

Those answers are provided by tools used for TCP Root Cause Analysis. While some factors are obvious, others need further investigation. However, identifying possible causes that explain the observed throughput at a given time instant is non-trivial when traffic can only be observed at a particular point in the network without accessing e.g the client host.

TCP’s throughput may be limited for several reseaons. A server might not have enough bandwidth to saturate the users’ access line, or a link connecting the user to the server might be bottlenecked and thus limit the throughput. High load in the network can cause congestion and a TCP sender will limit its sending rate in congestion avoidance phases. Furthermore, the sending rate might be simply application limited, e.g. in case of a voice over IP client which sends a small amount of data in very frequent intervals and thus the application will not attempt to use all of the available network resources. The latter is a major cause for a throughput limitation (Siekkinen, PAM 2007).

While most of the rate limiting factors cannot be controlled by the users, some can. The amount of unacknowledged data that can be outstanding at any time is defined by the TCP window. Assume the window size is 64 Kb and the round trip time (RTT) is one second. Then his will result in 64 kb that can be transferred per second. Thus, also the RTT, that will increase with increasing distance to the remote server, can be a limiting factor. Moreover, if the window size is low, either at the sender (sender window) or at the receiver (receiver window), the application will also experience low throughput. In contrast to other limiting factors, the size of the (advertised) receiver window can be directly controlled by the user and a misconfiguration might be one possible cause of low throughput. The pioneering work by Zhang et al. found that congestion and limited windows are common causes for low throughput in observed TCP connections and showed that congestion is not always the cause for throughput limitations. Therefore, the view that throughput is limited by the network only is too restrictive.

Some root causes can be identified using a nice online tool. It allows to quickly get some TCP connection statistics (like RTT and congestion window measurements) for performing a TCP Root Cause Analysis by simply accessing a web page. The tool can be accessed here. An extended tool is proposed by Siekkinen et al.

Readers interested in TCP Root Cause Analysis should consider reading the following papers:

• M. Siekkinen, G. Urvoy-Keller, E. W. Biersack, and D. Collange. A Root Cause Analysis Toolkit for TCP. Computer Networks, 52(9):1846–1858, 2008.
• Y. Zhang, L. Breslau, V. Paxson, and S. Shenker. On the Characteristics and Origins of Internet Flow Rates. ACM SIGCOMM 2002.

The last IETF meeting was held in Philadelphia last week. Recall my previous posting about turning the protocol stack upside-down. Jonathan Rosenberg, who is with Cisco, presented a new trend in a talk at the meeting: TCP over UDP. The main motivation for this approach is NAT traversal, which can be done using UDP pretty well. Unfortunatly, the UDP protocol has, among other things, a lack of flow control, which is a pretty compfortable thing application level programmers do not want to miss. The combination of both advantages (NAT traversal and e.g. flow control) results in tunneling TCP over UDP. Thus, there is one more best effort datagram protocol layer in between. Does this endeavour highlights the need for a new Internet, as researcher in the Future Internet sector are claiming?

I discovered RFC 3251 today, which describes Electricity over IP as persiflage to numerous RFC’s published by the IETF themed “X over Y”, whereas “X over Y” will be followed by “Y over X” sooner or later. Just for instance, once we had the usual setup where IP packets were encapsulated in Ethernet frames which is a often used link layer protocol (IP over Ethernet). Nowadays, Ethernet over IP is possible as well (RFC 3378: “EtherIP: Tunneling Ethernet Frames in IP Datagrams“), turning the ordinary protocol stack upside down by sending layer 2 protocols over layer 3.

Once upon a time, MPLS was used to go round the “time intensive” routing process in IP backbone networks by using pre-defined switching paths to guide the packet’s way through the network (IP over MPLS). Since June 2007 we got RFC 4817 describing how to encapsulate MPLS in IP packets (MPLS over IP).

In the 90’s, service providers were deploying ATM networks before they were suppressed by the more cost-effective Ethernet technology about a decade later [1]. ATM was once famous partly due to its Quality of Service abilities. Following the general trend, the next step towards a upside down protocol stack is to introduce the notion of pseudo-wires in order to route ATM over IP.

The next step is to tunnel Synchronous Optical Networking (SONET) over MPLS (over IP over … ?), which is interesting as SONET is synchronous whereas IP is not and easily leads to jitter within the transmission.

By the way, not only the IETF is tunnelling the whole protocol stack from every possible direction when needed. Also amateur radio operators encapsulate TCP/IP in AX.25 (the layer 2 protocol used for wireless transmissions in amateur radio) using the 44. class A IP subnet for accessing their own WWW called HamWeb or chatting over IRC instead of more appropriate solutions. So here we have IP over AX.25, which is just like IP over Ethernet, fine. However, when no radio is available, the radio network can be accessed by tunnelling AX.25 over IP (AXIP). So here we’re sending a layer 2 protocol over a layer 3 one again.

As Spam over IP is already old-fashion, we now got IP over Spam. Yes, online banking can now be encapsulated in Viagra! Note that this approach is new as layer 3 packets are now encapsulated in layer 7 data! We do no longer stick to tunnelling the lower layer but consider higher layers also. The potential benefit of this encapsulation is that the great (fire)wall in China can be overcome, as they’re sending and receiving a lot of spam which turns it into a high-bandwidth, low-latency channel, as Dan Kaminsky highlighted in his talk.

So what’s next? Well, RFC 3251 will give the definite answer: Electricity over IP, where IP packets carry electricity in discrete, digitalized form. The document is based on the discovery that the distribution network for electricity is not an IP network. So do service providers have to extend their triple play offerings (classical Internet, (IP)TV, telephone) by electricity over IP to get the IETF into non-technical areas such as the distribution of electricity? Electricity could be routed “over the Internet to reach remote places which presently do not have electricity connections but have only Internet kiosks (e.g., rural India)”. Well, I suggest to read the document and find out about newly emerging technologies, such as:

• MPLampS: Mostly Pointless Lamp Switching
• LER: ‘Low-voltage Electricity Receptor – fancy name for “lamp”‘
• VPN: Voltage Protected Network

So long,

Oliver — waiting for the first electricity trojan or worm and wondering whether houses are no longer in danger due to electricity thieves.

PS: When will we become tunnelling approaches that consider layer 8 (guess who’s controlling the applications …)? I could imagine IP over long or short term memory by memorizing the packet. I’m wondering how Raptor codes (a state-of-the-art Forward Error Correction technique) would perform when coping with an error process called forgetting?

PPS: RFC 2549 introduced Internet Protocol over Avian Carriers (IPoAC) . Will there be Avian Carriers over IP soon?

[1] Disclaimer (just to prevent of someone getting me wrong): Yes, I know how ADSL traffic is still carried (although there is a trend to migrate to Ethernet as this technology is cheaper than ATM due to its wide-spreadness), remember that this post is devoted to RFC 3251 which is a joke so some statements may be a bit “overgeneralised” here Moreover, I also know they benefits of tunnelling layer 2 protocols over IP, they are mostly written in the article “Layer 2 over IP/MPLS” by Chris Metz (Cisco) published in IEEE Internet Computing journal back in 2001. However, this post is not an article discussing pro’s and con’s of several technologies.