B.Comp. Dissertation
Towards Universal Connectivity
By Yang Qingwei
Department of Computer Science School of Computing National University of Singapore 2009/10
B.Comp. Dissertation
Towards Universal Connectivity
By Yang Qingwei
Department of Computer Science School of Computing National University of Singapore 2009/10
Project No: H116160 Advisor: Dr. Ben Leong
Deliverables: Report: 1 Volume Source Code and API: 1 CD
Abstract Over the past decade, we have seen the introduction of many new Internet appliances and services. The huge growth in the number of Internet users and the rising number of wireless mobile devices has led to the increase in demand for IPv4 addresses. Given the limited number of IPv4 addresses available, several techniques have been deployed to maximise the use of its address space. One of the techniques used to extend the lifetime of IPv4 address space is to deploy Network Address Translators (NATs). NAT allows multiple devices to access the internet through a single public IPv4 address. Due to its usefulness, NAT grew in popularity and became a standard feature in today’s routers. The disadvantage of NAT is that it hinders the establishment of connection between two end-devices. As a result, any peer-to-peer (p2p) applications today will almost certainly require NAT traversal in order to establish end-to-end connectivity. While there are many NAT traversal solutions available, existing solutions are either cumbersome to set up or costly. These solutions require the provisioning of dedicated servers to aid in the traversal attempt and to act as NAT proxies when NAT traversal fails. If developers are presented with an alternative solution that is simple and not as costly, we will significantly lower the barriers of entry for novel p2p applications. The goal is to develop a simple interface for developers to create new classes of applications that can allow any two devices connected to the Internet to communicate, including personal computers, laptops, and mobile phones. Our solution consists of two parts: i) a set of client network libraries, and ii) a shared NAT traversal system deployed “in the cloud”. The complexities associated with NAT traversal will be hidden from the developers through our network library. Subject Descriptors: C.2.1 Network Architecture and Design C.2.4 Distributed Systems C.4 Performance of Systems Keywords: Distributed systems, computer networks, peer-to-peer, network address translator Implementation Software and Hardware: Windows Platform, Java SE 1.6, Eclipse IDE, Apache, PHP
Acknowledgement
I would like to thank my project supervisor, Dr. Ben Leong, for his constant supervision and help rendered during the progress of this project and the writing of this report. Indeed, without his tutelage, my project would not be at its current state of completion and polish.
Dr. Leong was involved at all stages in the development of the project. He made a point to ensure that I had the benefit of a holistic learning experience with regards to projects and project management. He often reminded me about the other aspects of project management aside from the technical aspect that I am familiar with. A few of such aspects include human resource, time and goal management.
Lastly, Dr. Leong believed very strongly in providing accurate and honest feedback when it was due. Though painful to stomach initially, I learned that such chastisements were eventually for my own good when I saw the positive effect of changing my wrong methodologies come to pass.
I would also like to take this opportunity to thank Leong Wai Kay, Daryl Seah, Ali Razeen, and Lim Yuen Hoe for their constant support, valuable feedback and assistance in the design and development of this project.
List of Figures Figure 1.1: Network Address Translator ................................................................................... 1 Figure 3.1: Overall System Architecture Diagram .................................................................. 10 Figure 3.2: Identity Resolution Service ................................................................................... 12 Figure 3.3: NAT Traversal Service.......................................................................................... 13 Figure 3.4: Peer Proxy Support ................................................................................................ 13 Figure 3.5: Mobile Proxy Service ............................................................................................ 15 Figure 4.1: Application Flow Diagram .................................................................................... 19 Figure 4.2: Identity Registration and Management Service .................................................... 20 Figure 4.3: Web Management System (www.OpenNAT.com) ............................................... 22 Figure 4.4: Server-to-Server Hole Punch Coordination .......................................................... 23 Figure 4.5: Hole Punching Example ........................................................................................ 23 Figure 4.6: Peer-proxy Example .............................................................................................. 24 Figure 4.7: Server-proxy Example ........................................................................................... 25 Figure 4.8: Peer Proxy Departs from Network ........................................................................ 26 Figure 5.1: Karkinos Facebook Game ..................................................................................... 27 Figure 5.2: Karkinos Architecture ........................................................................................... 28 Figure 5.3: Time taken to establish connections ...................................................................... 30 Figure 5.4: Finding a peer proxy.............................................................................................. 32 Figure 5.5: Simulated connection times for proxied UDP connections................................... 33 Figure 5.6: Simulated connection times for proxied TCP connections ................................... 33 Figure 5.7: Simulated maximum throughput for proxied UDP connections ........................... 34 Figure 5.8: Degree of Separation for Various Network Sizes ................................................. 34 Figure 5.9: Round-trip latencies for direct and proxied UDP Connections ............................. 35 Figure 5.10: Round-trip latencies for direct and proxied TCP connections ............................ 36 Figure 5.11: Throughput for direct and proxied UDP connections ......................................... 36 Figure 5.12: Throughput for direct and proxied TCP connections .......................................... 37 Figure 5.13: Loss rates for UDP connections .......................................................................... 38 Figure 5.14: Experimental Setup for 3G Mobile-to-Mobile .................................................... 39 Figure 5.15: Mobile Carrier Network ...................................................................................... 40
iv
List of Tables Table 3-1: IdentityService Class Method Summary ................................................................ 11 Table 5-1: OpenNAT Hole punching statistics........................................................................ 29 Table 5-2: RakNet Hole punching statistics ............................................................................ 30 Table 5-3: UDP and TCP Direct Connection .......................................................................... 39 Table 5-4: Cross-carrier UDP NAT Traversal ......................................................................... 40 Table 5-5: Cross-carrier TCP NAT Traversal ......................................................................... 41 Table 5-6: UDP DatagramSocket Initialization Comparison .................................................. 41 Table 5-7: TCP ServerSocket / Socket Initialization Comparison .......................................... 41 Table 5-8: Sending DatagramPacket Comparison ................................................................... 42 Table 5-9: TCP OutputStream Comparison............................................................................. 42 Table 5-10: Receiving DatagramPacket Comparison .............................................................. 42 Table 5-11: TCP InputStream Comparison ............................................................................. 43
v
Table of Contents Title............................................................................................................................................. i Abstract.....................................................................................................................................ii Acknowledgement .................................................................................................................. iii List of Figures.......................................................................................................................... iv List of Tables ............................................................................................................................ v 1. Introduction ...................................................................................................................... 1 1.1 1.2 1.3 1.4 1.5
The Problem ................................................................................................................ 1 Our Solution ................................................................................................................ 3 Potential Impact of Our Solution ................................................................................ 3 Scope of Project .......................................................................................................... 4 Report Organization .................................................................................................... 5
2. Related Work .................................................................................................................... 6 3. System Design ................................................................................................................. 10 3.1 3.2 3.3 3.4
System Architecture .................................................................................................. 10 OpenNAT Network Library ...................................................................................... 11 OpenNAT Cloud ....................................................................................................... 13 Key Challenges ......................................................................................................... 15
4. System Implementation.................................................................................................. 18 4.1 4.2 4.3 4.4
OpenNAT Network Library ...................................................................................... 18 Identity Resolution and Management ....................................................................... 21 Hole Punching ........................................................................................................... 22 Proxy Support............................................................................................................ 24
5. Evaluation ....................................................................................................................... 27 5.1 5.2 5.3 5.4
Experimental Setup ................................................................................................... 28 Results ....................................................................................................................... 29 Preliminary Study on 3G Mobile Phone-to-Phone Connections .............................. 38 Code Comparison between OpenNAT and JavaSE .................................................. 41
6. Conclusion ....................................................................................................................... 44 6.1
Future Work .............................................................................................................. 45
References ............................................................................................................................... 47 Appendix A – UDP Chat Application Example .................................................................. 50
Chapter 1
1. Introduction Over the past decade, we have seen the introduction of many new Internet appliances and services. The usage of the Internet increased by almost four times, from 456 million users in the year 2000 to 1.734 billion in 2009 (Internet World Stats, 2009). This trend is expected to continue as the availability and usage of wireless mobile devices increases steadily. For instance in 2008, laptop sales exceeded desktop sales on the global level for the first time (Laptoplogic.com, 2008). Worldwide sales of smartphones have also increased by 13.9% (FierceDeveloper, 2009) and are expected to account for more than 30% of the Internet Traffic in 2013 (Patel, 2009). This huge growth in the number of Internet users and the rising number of wireless mobile devices has led to the increased in demand of IPv4 addresses.
1.1 The Problem The exponential increase in the use of the Internet has raised concerns over the depletion of the IPv4 address space. In 1992, the Internet Engineering Task Force started designing a new Internet Protocol called IPv6 to address this raising concern (Deering, 1998). As an interim solution, the Network Address Translator (NAT) technique was developed to extend the lifetime of IPv4.
Figure 1.1: Network Address Translator
1
NAT allows multiple clients to connect to the Internet using a single public IPv4 address (Figure 1.1). It works by assigning each client a private IPv4 address and by keeping track of the Internet connections each client has initiated with some destination. NAT has proven to work effectively in practice and has become a standard, indispensable feature in today’s routers. Unfortunately due to the way NAT works, these NAT devices will usually reject incoming connections that the client did not initiate. This behaviour can be configured by setting up Universal Plug and Play (UPn
P) (International Organization for Standarization, 2008)
or port forwarding where clients can specifically allow incoming connections on certain ports. However, it is not always the case that users have access to the router to perform these configurations and it is also not clear that users will know how to perform such configurations correctly. As such, users with mobile devices and personal computers that are behind NAT no longer have direct end-to-end connectivity with some other devices on the Internet. This makes the deployment of peer-to-peer (p2p) applications like mobile phone-to-phone video streaming, file sharing, and p2p games significantly more difficult. There is a need to perform NAT traversal to achieve end-to-end connectivity. Application developers can turn to several available libraries to perform NAT traversal but these require the provisioning of dedicated servers that aid in the traversal attempt. These libraries may also not be compatible across applications and platforms and uniquely identifying connected clients becomes a non-trivial task. The reason is because various libraries allocate identities in their own way and such identities might not be consistent between different sessions. In addition, existing NAT traversal techniques also rely on NAT hole punching which works only about 90% of the time (D. Seah, November 2009). When NAT hole punching fails, two clients will have to communicate through a proxy. Supporting NAT traversal is therefore complicated and costly because it involves not only setting up dedicated, publicly-reachable NAT servers, but also proxy servers to support clients when NAT hole punching fails. Although IPv6 became publicly operational in 2003, a widespread transition to IPv6 has not taken place (PCWorld, 2008). We believe that the ubiquity of NAT devices is the main 2
reason why this has not happened and make a bold prediction that the cost and inconvenience in migrating fully to IPv6 will continue to deter transition for at least another decade. Even if IPv6 is eventually rolled out, NAT will be required during the transition phase to connect between IPv4 and IPv6 networks. In addition, the Chairman of the Internet Engineering Task Force – the designers of IPv6 – said that NATs would still be necessary after the full transition to IPv6 (Carolyn Duffy Marsan, 2008). Most enterprises also demand NAT devices for IPv6 in order to preserve private IP addressing plans and to conceal their network topologies (NetworkWorld, 2009). Our goal therefore is to develop a universal platform to support and provide developers with a simple way of overcoming the problem of establishing direct connections between NAT'ed clients, and lower the barriers to entry for novel p2p applications.
1.2 Our Solution OpenNAT is a simple platform for developers to create applications that are universally connectable across devices such as personal computers, laptops and mobile phones, through a shared NAT traversal system deployed “in the cloud”. Our solution consists of two parts: i) a set of client network libraries implemented in C, C#, Java, Android and iPhone platform, and ii) the OpenNAT Cloud. The OpenNAT Cloud comprises of three key services: Identity Resolution and Management Service, NAT Traversal Service with Proxy Support, and Mobile Proxy Service. Details of our solution will be further discussed in Chapter 3 and 4.
1.3 Potential Impact of Our Solution The opportunity to lower the cost and improve the overall quality of network applications drives the potential impact of OpenNAT. We posit that in the near future, more developers will see the benefit of using OpenNAT services when implementing network applications. For a small subscription fee, developers will no longer have to deal with the complexity of setting up and maintaining additional servers to support NAT traversal in their applications. This will allow them to minimize extended server support and move these resources into other projects. End-users on the other hand, would also benefit from a more seamless and 3
enjoyable experience as they no longer have to fuss over router and firewall configurations when using developers’ applications. They would also not need to know their external public IP address even if they wish to host a server in a private network. A simple, easy to remember identity scheme will be provided for connections instead. Overall, we feel that OpenNAT is a revolutionary solution to the current problems that many developers face when attempting to implement p2p applications.
1.4 Scope of Project The objective of this project is to develop the OpenNAT system consisting of NAT Traversal Service with proxy support, Identity Resolution and Management Service and a client network library in JavaSE. The design and implementation of the Mobile Proxy Service and remaining platform of network libraries (C, C#, Android and iPhone) are not within the scope of this project. As mentioned in the Introduction, two clients will have to communicate through a proxy if NAT hole punching fails. Hence to aid us in the implementation of this project, we have to answer three key research questions: i) Are NAT peer proxies easy to find and if so, what is the performance? ii) How can we find a peer proxy with good performance? iii) How does our implementation fare against other NAT traversal libraries? We demonstrated with a measurement study that using peers as NAT proxies is feasible for both UDP and TCP connections. We found that it is relatively easy to find peers capable of acting as proxies and that the performance achieved is comparable to that of server-based NAT proxies. We also found that our simple proxy selection scheme of broadcasting requests to five peers and selecting the first respondent as the proxy is sufficient to provide good performance. Lastly, we evaluated our variation of NAT hole punching with a commercialgrade game network library, RakNet (Jenkins Software, 2010). We found that RakNet’s NAT traversal timeout settings may be too short as our results indicated that NAT traversal may take longer than 10 s. We submitted a research paper detailing the results of our experiments and it was accepted for publication at the 8th International Workshop on Network and Systems Support 2009 (D. Seah, November 2009). 4
We also wanted to know the behaviour of mobile phone-to-phone connections so as to facilitate future implementation of the mobile platform library. Hence, we conducted a preliminary study on Singapore’s 3G mobile carrier networks to have a basic understanding of their NAT infrastructure. The details of all our findings will be further discussed in Chapter 5. Lastly, we developed a chat application to demonstrate the simplicity of establishing end-toend connectivity using the OpenNAT platform. The application is capable of establishing a connection with any hosts regardless of the type of network they are in.
1.5 Report Organization The remainder of this report is organized as follows: in Chapter 2, we provide a review of existing and related work. We then describe the design of the OpenNAT architecture in Chapter 3 and the details and interesting aspects of our implementation in Chapter 4. Finally, we provide results of our experiments and evaluation in Chapter 5 before concluding in Chapter 6.
5
Chapter 2
2. Related Work NAT traversal for UDP was pioneered by Dan Kegel and his techniques have since been standardized to become the STUN protocol (Simple Traversal of UDP through NAT) (J. Rosenberg R. M., 2008). STUN works by initiating a short sequence of requests to an external server. The server will then respond with network information such as the public IP address and port binding properties associated with the application’s network connection, enabling an application to learn about the behaviour and operating methods of the NAT. Depending on the type of NAT and topological conditions of the network, the application may be able to predict the next mapped address and port that the NAT will assign. This enables two applications to initiate a connection with each other’s mapped address and port even though the mapping is allocated by the NAT after the connection is initiated. This NAT traversal process is commonly known as ‘hole punching’. Guha et al. subsequently developed STUNT (STUN and TCP), which extended STUN to include port prediction for TCP NAT traversal (S. Guha Y. T., 2004).
As NAT traversal with STUNT is not always successful, the TURN protocol (Traversal Using Relay NAT) (J. Rosenberg R. M., 2005) was developed to use a proxy node to relay data should hole punching fail. While TURN almost always succeeds in providing connectivity to a client, it can put a heavy load on the proxy server. Therefore it is desirable to use TURN only as a last resort, preferring other mechanisms such as STUNT when possible.
A number of attempts have been made in recent years to evaluate the performance of these protocols. In 2005, Ford et al. distributed a test program called NAT Check, and solicited data from Internet users about their NAT. They found that 82% of the users’ NAT supported UDP hole punching while 64% supported TCP hole punching. They concluded that while the 6
data collected does not represent the true distribution of the NATs in common use, it appears that the majority of commonly-deployed NATs already support UDP and TCP hole punching. At around the same time, Guha et al. evaluated the effectiveness of STUNT both with a lab test bed and in the wild. Their study showed that although there are a significant number of problems associated with TCP NAT traversal, they were still able to achieve a success rate of about 88% (S. Guha P. F., 2005). They also measured the time taken to establish a TCP connection, but only for a single attempt without retrying in the case that port prediction failed. We have repeated some of these measurements and also measured the time taken to perform NAT traversal for both UDP and TCP.
Some studies have also centred on Skype, a popular VoIP application that uses peers to relay VoIP traffic. Measurement studies by Guha et al. showed that the behaviour of nodes in the Skype network differs significantly from other p2p networks (S. Guha N. D., 2006). They found that supernodes (i.e. potential relays) are relatively stable when compared to nodes in other p2p networks and that the bandwidth consumed by an active relay is fairly low. Another study by Kho et al. classified the relay nodes by their domain name suffixes (W. Kho, 2008). They found that 22.4% of the relay nodes end in .edu, suggesting that the Skype network might be sustained by high-bandwidth non-NAT university networks. Their study also showed that the uptime of Skype’s relay nodes follow a diurnal pattern and that its selection mechanism can be improved by using techniques such as network coordinates. Lastly, Ren et al. also investigated the performance of the Skype system and found that the selections of relay nodes are suboptimal (S. Ren, 2006). They observed that the waiting time required to select a node can be quite long and that there were a large number of unnecessary probes, resulting in heavy network traffic that limits scalability of the VoIP system. As a result, they proposed ASAP, an AS-aware peer relay protocol for VoIP, suggesting that one-hop peer relaying can provide higher quality VoIP transmission when compared to direct IP routing. These works suggest that the benefits of relaying are not limited to circumventing NAT and its effectiveness depends on the nature of the application and the demographics of the users in the network.
While there have also been proposals for finding and selecting appropriate peer relay nodes, previous proposals have only been simulated and have not been validated in “live” 7
deployments. Dowling et al. suggested the use of the gradient topology to help in locating good quality relay nodes in a generic p2p network (J. Dowling, 2007). The idea is for nodes in a p2p network to use a local quality level to adapt their connections to other peers, such that peers with the highest quality levels are clustered in the center of the topology while peers with decreasing quality are found at increasing distance from the center. Their study showed that the topology not only improves the time required to find a good quality relay node, but also help optimize the global utilization of resources in the p2p system. Nguyen et al. also proposed a relay selection algorithm to improve load balancing (H. X. Nguyen, 2008). The algorithm builds upon the classic load-based algorithm where they proved that it can effectively balance the load of relays even when they are heterogeneous, and that its performance depends directly on relay heterogeneity. They also showed with a simulated numerical evaluation that their proposed algorithm can effectively distribute the load in order to balance relay utilization, even when relay popularity and capacity are uncorrelated. To the best of our knowledge, there has not been a comprehensive study on the use of peers as relays for NAT traversal.
There are also existing libraries available for developers to create p2p networked applications, and most of them have some form of NAT traversal support. Some of the more popular libraries are summarized below.
i) Sun Microsystems developed JXTA (Li, 2001), which uses XML messages to support communication between devices in a way that is independent of network topology. Version 2.0 introduces super-peer relays to bridge peers that do not have direct connectivity and supports transparent re-routing if the relay fails (Bernard Traversat, 2003). However, to the best of our knowledge, JXTA does not support STUNT or other hole punching techniques.
ii) Microsoft’s Windows Communication Foundation (Chappell, 2007) also provides p2p support for applications like peer naming and resolution, and also a communication channel between peers. NAT traversal is only supported with a NAT API that employs Universal Plug-and-Play (UPnP) to configure the NAT devices. To the best of our knowledge, no other techniques are built into the framework. 8
iii) Extensible Messaging and Presence Protocol (XMPP) (Saint-Andre, 2004) is a middleware layer that was developed as the communication infrastructure for Jabber Instant Messenger. While it started out as a means to handle p2p messaging and file transfer, it has evolved into a development framework adaptable for general use. XMPP provides for transportation using packets over UDP and byte streams using TCP. NAT traversal methods such as STUN are implemented for UDP and proxy service for TCP is provided. However no hole punching methods for TCP are documented.
iv) RakNet (Jenkins Software, 2010), the networking library used by Demigod (Gas Powered Games, 2009), is by far the most popular library among game developers for traditional server-client games. P2P support has been recently added as plug-ins that support NAT traversal via hole punching and proxying. In view of its commercial popularity, we evaluate the performance of RakNet’s NAT traversal in addition to our library.
Lastly, the difficulties of identifying and reaching end-users on mobile devices was first highlighted by Bryan Ford, who proposed UIA (Unmanaged Internet Architecture), a p2p architecture that allows the addressing of users and their devices via personal names (Ford, 2008). UIA works in a decentralized manner and focuses on distributed name assignment and resolution. It also attempts to route around link failures to maintain uninterrupted connectivity. However, UIA assumes the existence of peers which are not behind NATs or firewalls for routing data, and will ultimately need supplementary servers if such peers do not exist. Our approach employs cloud resources to support the system, because we believe that it is a more scalable and practical approach.
9
Chapter 3
3. System Design In this chapter, we begin by introducing the overall system architecture of OpenNAT. We then describe the design of the client library and server component before articulating the key challenges of building OpenNAT.
3.1 System Architecture The architecture of OpenNAT is illustrated in Figure 3.1. The OpenNAT network library is the main component used for interaction with OpenNAT services in the cloud. Developers will use the library to build network applications capable of interacting with each other regardless of the nature of the network and type of device.
Figure 3.1: Overall System Architecture Diagram
10
3.2 OpenNAT Network Library The network library uses the OpenNAT Cloud as a rendezvous point (Figure 3.1). The servers manages and distributes peer identifiers, coordinates NAT hole punching and acts as a proxy server when necessary. The library provides control signalling between peers and the servers similar to the Session Initiation Protocol(J. Rosenberg H. S., 2002) to maintain the overlay peer network. The library is developed in JavaSE and is designed to provide the DatagramSocket, Socket and ServerSocket interfaces to the developer. These interfaces are similar to the ones that the standard Java libraries provide. Therefore, developing universally connectable p2p applications are simple as developers would be familiar with the original interfaces. Existing client-server applications can also be easily converted to p2p using the OpenNAT network library. In Chapter 5.3 of Evaluation, we show a comparison between the source-code of a simple chat application implemented using the OpenNAT library and standard JavaSE libraries. The API of the OpenNAT library can be found in the CD included with this report.
3.2.1 Identity Resolution Service Because NAT allows multiple hosts and its applications to share a public IP address, there is a need to identify and differentiate them. In the OpenNAT system, applications are identified with an easy to remember network-independent address called identity, in a flat name space similar to the domain name space. These identities enable the applications to be contactable by other applications. Users of the applications will no longer have to fuss over finding and remembering their external public IP address in a private network, especially if they wish to host a server. Examples of identities are ‘Qingwei.P2P’ and ‘Qingwei.MOBILE’. IdentityService Class static
IdentityService boolean void Node
getInstance() register(String identity) unregister() lookup(String identity)
Table 3-1: IdentityService Class Method Summary
11
Table 3-1 shows the method summary of the IdentityService class. The class uses a singleton design pattern as only one instance of the object is required. Applications must be registered with the OpenNAT Cloud before it can use the OpenNAT services. This can be easily done by calling the register() method.
Figure 3.2: Identity Resolution Service
In Figure 3.2 for example, user A is registered with the OpenNAT Cloud with the identity “Qingwei.MOBILE”. To connect to user B, user A queries the OpenNAT server for the network information of identity “Qingwei.DESKTOP” (Step 1). The OpenNAT identity server resolves the requested identity (Step 2), and returns a Node object containing network information of User B (Step 3). With this information, user A will be able to establish a direct connection to user B using the OpenNAT socket interfaces.
There are two kinds of identity schemes in OpenNAT: i) custom identities for paidsubscribers, and ii) custom ad-hoc identities for non-subscribers. Paid-subscribers will be able to register custom identities and define a limited number of sub-identities for a small fee. For non-subscribers, custom ad-hoc identities can be created but it comes with a fixed prefix (e.g. ‘@’) added to the requested identity (e.g. @Identity.SubIdentity) and has a limited lease time. When the lease time expires, the application will no-longer be identifiable using the identity. The application will have to re-register with the OpenNAT Cloud in order to be contactable again. 12
Figure 3.4: Peer Proxy Support
Figure 3.3: NAT Traversal Service
3.2.2 NAT Traversal with Proxy Support Our solution ensures connectivity between two peers by transparently applying various NAT traversal techniques such as STUN (J. Rosenberg R. M., 2008) and STUNT (S. Guha Y. T., 2004) (Figure 3.3). In the event that no direct connection can be established, the network library will transparently fall back to relaying through a server or a fellow peer, for both UDP and TCP connections (Figure 3.4). Our TCP proxy method also provides the flexibility to dynamically reroute data through a different proxy without breaking the connection. This allows us to use an interim proxy while performing NAT traversal to prevent delays.
3.3 OpenNAT Cloud The OpenNAT Cloud comprises of three services: i) Identity Resolution and Management Service, ii) NAT Traversal Service with Proxy Support, and iii) Mobile Proxy Service.
3.3.1 Identity Resolution and Management Service The Identity Resolution Service is responsible for managing identity registration and resolution of the clients’ network information (Figure 3.2). An online management system in the form of a web service is also provided to support the management of permanent (paid) identities and also ad hoc identities for non-paying 13
subscribers. Users will be able to register their identities (e.g. Qingwei) and manage subidentities (e.g. Qingwei.MOBILE, Qingwei.LAPTOP), similar to the concept of subdomains. Additional features such as analytics could be added to provide users with useful information about their identities in the future.
3.3.2 NAT Traversal Service with Proxy Support The NAT Traversal Service is supported by a set of centralized servers that assist clients in establishing direct connections (Figure 3.3). The service serves as a signalling server to help peers coordinate a hole punching attempt. In the event that no direct connection can be established, the service will first assist clients in searching for a peer proxy (Figure 3.4). If no suitable proxies can be found, the OpenNAT server will function as the proxy for the clients.
3.3.3 Mobile Proxy Service The goal of the Mobile Proxy Service is to provide a virtual persistent connection to mobile devices that is independent of the underlying network connectivity. The service acts as a virtual end-point for the device, which is persistent and always reachable (Figure 3.5). When connectivity to the mobile device is temporarily lost, data will be buffered in the proxy and transmitted when the connection is re-established. When the connection mode of a mobile device changes, for example from 3G to WiFi, the virtual end-point on the proxy remains unchanged. This allows the user’s online presence to remain persistent even as the mobile phone negotiates different modes of connectivity to the Internet. In addition, the proxy can accept incoming connections from other devices on behalf of the actual device. This will provide mobile users with a seamless experience for connection-based applications.
14
Figure 3.5: Mobile Proxy Service
3.4 Key Challenges A key design goal of OpenNAT is scalability. We would like the system to function correctly even in the presence of millions of hosts. We have identified several key challenges that have to be addressed to ensure smooth operation.
3.4.1 Identity Management The first key issue has to do with Identity Management. Due to the possibly large number of requests, two clients may simultaneously register for the same identity. Hence, there must be a mechanism in place to prevent such conflicts in the namespace. Identities should also be structured in a way that facilitates quick lookup, updating and validity checking.
3.4.2 Security Next, the identity resolution service provided by OpenNAT is similar to Domain Name Systems (DNS) (Postel, 1994). The security shortfalls of DNS such as cache poisoning (Klein, 2007) have to be addressed to maintain the integrity of OpenNAT services.
Identities that are paid-for and owned by users also have to be managed securely. The online identity management system has to incorporate a suitable authentication scheme and transmission protocol to prevent attacks such as phishing (Rachna Dhamija, 2006) and man-in-the-middle attack (N. Asokan, 2005).
15
3.4.3 Ease of Use The network interfaces of OpenNAT library should closely follow the standard of the respective programming platforms. This will reduce the learning curve and encourage developers to leverage on OpenNAT to develop novel p2p applications.
The user interface of the online identity management system should be simple and easy to use. At minimum, it should enable users to perform identity registration and revocation easily. Additional features such as analytics and command-line tools could be added to provide users with useful information and flexibility in managing identities.
3.4.4 Scalability Existing NAT server solutions are typically provisioned for several hundred users. OpenNAT is expected to have to support perhaps million of hosts and thus need to be incrementally scalable and robust.
The nature of p2p architectures and NAT traversal requires UDP sessions between the client and the OpenNAT server be kept alive. The server acts as a signalling server to help peers communicate with each other, resolve identities and look for suitable proxies. Hence, server performance may be an issue when there are many users. This problem can be easily addressed by dynamically scaling resource capacity in the cloud and applying appropriate load balancing schemes. Unfortunately, having multiple OpenNAT servers makes peer management and NAT traversal more complicated. For example, when a peer that is connected to server A wishes to communicate with another peer in server B, there must be some form of communication between the servers in order to communicate the peer’s existence and to coordinate a NAT traversal attempt. An appropriate protocol has to be designed to support NAT traversal coordination across servers.
3.4.5 Quality of Service When NAT traversal fails, peer proxies will be used to guarantee connectivity between clients. However, the key challenge of peers acting as proxies is that they might decide to leave the network unexpectedly. In such a scenario, it is possible for data to be sent 16
successfully to the proxy and yet be lost because the proxy departs before the data is forwarded to the destination. Hence, our proxies need to maintain sufficient state in order to support the semantics of the TCP/UDP abstraction correctly.
As hole punching takes time, a proxy should be used to relay data to the destination while it is taking place. Multiple connections to the same destination should also be multiplexed using an existing hole punched connection to avoid hole punching delay.
Lastly, a suitable proxy selection scheme has to be developed to maintain an acceptable level of service quality.
17
Chapter 4
4. System Implementation In this chapter, we discuss the implementation details of the OpenNAT Cloud. This chapter is organized into four parts, beginning with a description of the network library component. We then proceed to describe the Identity Resolution and Management, NAT hole punching, and Proxy support components.
4.1 OpenNAT Network Library For the scope of this project, the OpenNAT library is implemented on Java SE 6.0. The library provides support for direct communication between peers (through NAT hole punching), assigns unique identifiers to peers and provide proxy support. The library performs NAT hole punching when direct connection cannot be established and maintains the local connectivity to other known peers when hole punching is successful. Peer proxies or the OpenNAT cloud will be used as the last resort to relay data when NAT hole punching is unsuccessful. Figure 4.1 shows the general architecture of the OpenNAT library. A virtual socket is presented to the application for communication, which can be used with a socket policy to implement extra features like reliable datagram or even emulate
TCP
with
congestion
control. The virtual sockets are represented internally as different channels. The channels are then multiplexed to their respective native network sockets to be sent out on wire.
18
Figure 4.1: Application Flow Diagram
4.1.1 P2P Core The P2P core implements an overlay network with a new identity addressing scheme for clients connected to the OpenNAT Cloud (Figure 4.1 – P2P Core). It is responsible for managing the identities and network addresses of clients as well as communication with the OpenNAT Cloud. The overlay network enables users to retain their identity regardless of the IP address they are assigned. It also enables the library to dynamically change the port of the associated identities to match the new port that the NAT assigns in a different session. As a result, users will no longer need to know their external public IP address when they want to connect to other users or host a server in a private network. The implementation details of the interaction between the library and various services of the OpenNAT Cloud will be discussed in Chapter 4.2, 4.3 and 4.4.
4.1.2 Multiplexing Virtual Channels Two communication primitives – DatagramSocket and Socket interfaces – are supported by the library (Figure 4.1 – Interfaces). Due to the nature of NAT hole punching, the library will have to predict and use the port that will be assigned by the NAT boxes before a direct connection is made (Details in Chapter 4.3). This poses problems to application developers as they no longer have control over port binding. To address this shortfall, we introduced the concept of virtual channels to enable developers to define their desired ‘port’ in the form of channel identifiers. In our implementation, an 8-byte 19
header consisting of the source and destination channel identifier is added to each datagram packet sent. Packets that are received will be handed to the application based on the channel identifier instead of the original port that it came from. For Socket connections, each established connection will be mapped to a channel identifier. When the application requests for the input/output streams, the appropriate connection stream that correspond to the requested channel identifier will be returned to the application.
As the NAT hole punching process takes time to complete, the concept of virtual channels will enable us to take advantage of existing hole punched connections, if any. Data that is transmitted via the DatagramSocket virtual channels will be multiplexed and sent over a single, already existing connection. This reduces the connection time if multiple connections are required to the same destination. Multiplexing is not implemented for Socket connections as we do not want to disrupt the flow and congestion control that is present in TCP.
4.1.3 Policies Another advantage of virtual channels is that it provides the flexibility for developers to implement their own transmission policies such as Reliable DatagramSocket transmission or TCP over DatagramSocket transmission by extending the OpenNAT Socket class.
Figure 4.2: Identity Registration and Management Service
20
4.2 Identity Resolution and Management Before a peer can be contacted, it must first register itself with the Identity Resolution Service using the IdentityService.register() method. When the user attempts to register an identity, the Load Balancer will decide which Identity Resolution Server the peer should connect to (Figure 4.2). The requested identity will then be sent over UDP to the assigned Identity Resolution Service. If registration is successful, a Boolean variable with the value ‘true’ will be returned and the peer can now be contacted through the registered identity. A peer may also decide to unregister itself using the IdentityService.unregister() method. Whenever the peer is registered with the identity server, the library will periodically send keep-alive messages to keep the connection to the server active. Besides identity registration, a peer can also perform identity lookup by using the IdentityService.lookup() method to query the Identity Resolution Service. If the identity is currently registered, a Node object containing the id and network address of the peer will be returned to the peer and cached locally. A ‘shadow’ map is implemented in both the client library and service to facilitate reverse identity lookup using the Node object. The conventional approach of p2p architecture is to have every peer maintain a persistent connection with a tracking server so that any two peers can communicate to coordinate a hole punching attempt. This approach may not scale well when many peers are using the system. In our approach, the load balancing server is responsible for distributing peer connections evenly across the Identity Resolution Servers (Figure 4.2). It uses a simple scheme that distributes peer connections based on the number of peers connected to each Identity Resolution server. When a peer connects to or disconnects from the server, the information will be communicated to the load balancer and other identity servers in the network. All identity servers will keep a mirrored list of peers that are connected to the OpenNAT network. This will enable clients to lookup identities of peers located on other servers and establish direct connections to them.
21
Figure 4.3: Web Management System (www.OpenNAT.com)
We also implemented an online web management system to facilitate the management of paid-identities (Figure 4.3). The system incorporates a two factor authentication scheme and uses secure socket layer (SSL) for communication. To access the administration panel, users have to sign-on to the system using their registered username and password. Any changes made to the account such as adding or deleting of identities have to be verified with a onetime password that is sent to the user’s registered email account.
4.3 Hole Punching A variation of the standard hole punching techniques (B. Ford, 2005) with port prediction (S. Guha Y. T., 2004), including UPnP support and an augmented STUNT (S. Guha P. F., 2005) protocol for TCP hole punching was implemented.
To establish a UDP or TCP connection, we first check our internal cache for the destination peer address information. If the peer is not found, we will send an identity look-up request to the OpenNAT Identity Resolution Service to obtain the information of the peer. Once we know the network information of the destination peer, we will send an initial connection packet (as part of a 3-way handshake) to the most private address first. This is to ensure that we select the most efficient mode of communication if the destination peer co-exists in the same local area network. Simultaneously, the destination peer will be informed via the OpenNAT NAT Traversal Service to attempt to establish a reverse connection. If a 22
connection packet is received from the peer, the handshake is complete and the connection is established.
Figure 4.4: Server-to-Server Hole Punch Coordination
If no connection packet is received, two NAT Traversal servers will be queried in sequence (Step 1 and 2 in Figure 4.4) to obtain the external address and ports assigned by the peers’ NAT boxes. If both peers are connected to different NAT Traversal servers, the servers will communicate with each other and relay the information (Green arrow in Figure 4.4).
Figure 4.5: Hole Punching Example
With the external address and port information, we then predict the next port that the NAT will assign from the delta of the previous two port assignments (Figure 4.5). The destination 23
peer also performs this routine and both peers exchange their predictions via the signalling channel before attempting a connection with the predicted address/ports. If the NAT boxes assign the ports as predicted, a connection will be established. The process is repeated if the connection does not get through.
For both communication primitives, we will attempt direct and reverse connections on the known internal and external addresses of the peer first before doing port predicted connection attempts. This is to ensure that connection can be established quickly if the peers have no connectivity issues.
4.4 Proxy Support After several failed attempts at NAT hole punching, the OpenNAT library will attempt to create a proxy connection by using one of the available peers as the proxy. If no suitable peer can be found, the OpenNAT Proxy Service will be used as a last resort.
Figure 4.6: Peer-proxy Example
To select a peer proxy for UDP connections, the first packet will be sent simultaneously to a random set of 5 candidate peers obtained from the OpenNAT servers (Figure 4.6). The packet is then forwarded independently by these peers to the destination peer. The destination will respond with an acknowledgment to the first packet received and will drop subsequent duplicate packets. The peer that relayed this packet fastest will be chosen as the peer proxy. In our evaluation, we show that even though this scheme is simple, it can find a proxy with reasonably good performance with minimal delay. 24
Similarly for TCP peer selection, a packet will be sent using UDP to a random set of 5 candidate peers requesting for TCP proxy to the destination. The peers will each attempt a TCP connection to the destination and will inform the source of the result. The peer that sends the first successful acknowledgment to the source will be chosen as the proxy.
Figure 4.7: Server-proxy Example
When no suitable peer proxies can be found, server proxy will be used to guarantee connectivity between peers (Figure 4.7). This is possible because all registered peers have an active UDP connection to the server.
The key challenge in supporting peer-proxy is that peers can leave the network at any time. In other words, it is possible for data to be sent successfully to a proxy and yet be lost because the proxy departs before the data can be forwarded to the destination. Hence, our proxy needs to maintain sufficient state in order to support the semantics of our reliable UDP implementation and TCP abstraction correctly. To achieve this, we require all proxied data to be acknowledged by the destination host.
For our reliable UDP implementation, we add a unique message identifier to the header and record the average response time of each acknowledgement. After the transmission of the packet, we will store the proxied packet in a retransmission buffer. When the time exceeds the average acknowledgement time recorded, the library will resend the packet once more before starting the peer-proxy selection process if no response is received again. When a 25
suitable peer proxy has been found, retransmission will be made from the last acknowledged message identifier.
Figure 4.8: Peer Proxy Departs from Network
Our solution for TCP proxied connections is to split the data stream into segments, each delimited by a 7-byte header consisting of a delimiter, payload size, and segment and acknowledgement numbers. The headers are used to monitor and acknowledge the end-toend delivery of the segments as they are relayed by the proxy and reassembled at the destination. These segments are kept in memory at the source node until acknowledgments are received from the destination. When a TCP proxy leaves the network, a new proxy is selected and the two peers re-synchronize the data stream with the segments in the buffer (Figure 4.8). Because the underlying TCP streams are encapsulated from the application streams, the application streams are not broken and the application can continue the communication as if nothing happens, albeit a slight delay.
26
Chapter 5
5. Evaluation To evaluate the performance of our NAT traversal and proxy implementation, we developed a single player tower-defence game called Karkinos (A. Razeen, 2009), and deployed it as a Java applet via a Facebook application (Figure 5.1). The game is embedded with a test client that conducts the various p2p tests with other players playing the game. We wrote another test client in C++ using the RakNet library (Jenkins Software, 2010) to compare their performance against our implementation. We chose RakNet as it is arguably the most popular open-sourced commercial game networking library and has p2p features such as NAT traversal, proxy support and connection graph overlays. We also conducted a preliminary study on Singapore’s 3G mobile carrier networks to better understand their NAT infrastructure and the difficulties involved in mobile phone-to-phone connections. We present the results of our experiments in this chapter.
Figure 5.1: Karkinos Facebook Game
27
Figure 5.2: Karkinos Architecture
5.1 Experimental Setup Karkinos consists of two architectural components: the game applet with an embedded test client and the back-end services. Figure 5.2 shows the components of the game system, which consists of a web server to interface with Facebook, a measurement server, and a Global Tracker (GT). The GT functions as the rendezvous node that is queried when the game is first started. It is responsible for the pairing of clients and determining the order of experiments to perform. A persistent TCP connection is also made to the measurement server to submit network measurements. The Data Parser is used at the end of the whole experiment to process the data collected and generate the appropriate graphs with Gnuplot (Gnuplot, 2010). Our experiments were carried out over the Internet by volunteers who either played our Facebook game or ran our C++ RakNet test client. As our experiments were not conducted in a controlled environment, it is possible that data collected might be affected by the users’ Internet usage. For example, if one of the users was running a file sharing application at the same time, our experiments would likely be affected. However, we are not overly concerned because we are keen to get a snapshot of the network characteristics of a typical home user.
28
5.2 Results A total of 180 users played Karkinos and 94 users ran the RakNet test client over two separate sessions in October/November and July/October 2009 respectively. The maximum number of peers online at the same time was about 20 for both test clients. We have much fewer data points for RakNet compared to Karkinos because it seems that volunteers are more wary and less inclined to run a standalone Windows executable than a Java-based Facebook game. To ensure that the results are not skewed by users in our local university network, we filtered out such nodes from the data collected.
5.2.1 Connection Success Rates
Table 5-1: OpenNAT Hole punching statistics
In a separate experiment, we deployed another Facebook game called Tankie MAX and collected data on NAT hole punching success rates over a period of about 70 days in September 2009. We have included the results in this report as the game yielded more NAT hole punching attempts compared to Karkinos. The data collected is summarized in Table 5-1.
Our results are comparable to the findings by Guha and Francis in 2005 (S. Guha P. F., 2005). In addition, we also classified the attempted NAT connections by determining whether the endpoints were behind a NAT and whether they supported UPnP. It is not surprising that if one or more endpoints support UPnP, hole punching is generally more successful. However, we noticed for UDP, where if both NAT boxes support UPnP, hole punching performs slightly worse than if just one of them supports UPnP. At this point, we do not have a good explanation for this phenomenon. 29
Table 5-2: RakNet Hole punching statistics
RakNet’s native connection method assumes direct connectivity and does not perform NAT traversal. As such, UDP NAT hole punching is supported through a plug-in, which has to be explicitly invoked by the application. Unlike our implementation, there is no fixed protocol or NAT connection sequence for RakNet. To obtain a fair comparison, we developed the same sequence of events in the RakNet client when establishing a connection. When peer A initiates a connection to peer B and fails, the server will notify peer B to attempt a reverse connection to peer A; if this fails as well, peer A will initiate NAT hole punching using the hole punching plug-in at the default settings; if NAT hole punching fails, then the pair is deemed to be unconnectable. RakNet does not have its own UPnP implementation, and we were not able to get consistent results with a recommended third-party library and hence did not employ UPnP for the RakNet tests. Nevertheless, as shown in Table 5-2, the results are similar to that obtained with our Java-based library.
Figure 5.3: Time taken to establish connections
30
5.2.2 Connection Delays With the data collected from Karkinos and the RakNet client, we plot the cumulative distribution of the time taken to successfully establish various types of connections in Figure 5.3. UDP server-proxied connection times are omitted, because they are almost instantaneous. This result is expected because both RakNet and OpenNAT require applications to be connected and registered with the server. Since a connection has already been established, no NAT traversal will be required. The graph shows that NAT traversal can add significant delay to the connection time, with 10% of direct connections taking longer than 15 and 30 s for UDP and TCP respectively. The RakNet client appears to establish UDP connections faster than our implementation. We found that this is due to a combination of timeout settings in RakNet, which limits the time for any connection attempt to a maximum of about 10 s. Since our implementation shows that it may take longer than 10 s to successfully perform a UDP hole punch, it may be the case that RakNet’s default timeout settings are too short. We currently lack sufficient NAT traversal data for RakNet to provide a conclusive answer. It takes less time to create a connection through a server proxy for UDP and TCP than a direct connection because no hole punching is required to reach the server. While it may seem surprising that the establishment of a UDP peer proxy is also faster than a direct connection, this is due to the fact that requests are sent to several peers in parallel, which makes it more likely that the peers without NAT issues will fulfil the request earlier and reduce the overall connection time. Also, some proxies might have established the required UDP connections beforehand and so would not incur the time taken to perform NAT hole punching. On the other hand, setting up a TCP peer proxy can take exceptionally long, with about 20% of such connections taking longer than 30 s. This is expected though, as the process is more elaborate than setting up a UDP peer proxy. In addition, unlike UDP, NAT hole punching has to be performed for every new TCP connection request.
31
5.2.3 Finding Peer Proxies Since we are studying the feasibility of using peer proxies in a p2p setting, the two natural questions are: (1) how do we find suitable peer proxies and (2) are peer proxies readily available in a typical network? A trivial solution to the first question would be to send a message to every peer to check if it can be a proxy. While this might work in a small network, broadcasting to every peer in a large network would result in significant overhead and congestion.
Figure 5.4: Finding a peer proxy
To choose a proxy, our approach is to broadcast a proxy request to k randomly selected peers and choose the best peer after we have heard from at least l of them. If none of these peers are able to be a proxy, we repeat this process with another k peers until we exhaust all available peers and need to rely on a proxy server. To derive the optimal values for k and l, we collected data on the connection times, throughputs and latencies achieved between all pairs of nodes that were online at the same time and conducted simulations on the data set to obtain the estimated performance with different values of k and l.
32
Figure 5.5: Simulated connection times for proxied UDP connections
Figure 5.6: Simulated connection times for proxied TCP connections
Figure 5.5 and Figure 5.6 shows the average time taken to find a proxy in our simulations for both UDP and TCP respectively. Broadcasting to more peers decreases the connection time until k = 5, after which there is no significant improvement in increasing k. The graphs also show that waiting for more than one response (l) would increase the connection time significantly. To evaluate if it would be helpful to wait for additional responses, we plot the maximum throughput attainable amongst the proxies selected in Figure 5.7 for UDP. 33
Figure 5.7: Simulated maximum throughput for proxied UDP connections
While the average throughput appears to increase with l, the large spread of the data points suggests that the actual gain might not amount to much (Figure 5.7). We obtained similar results for TCP, which has an average throughput that is about 50 Kbps lower. The estimated minimum latencies between the chosen proxies do not vary significantly as well. In light of these results, we chose to broadcast proxy requests to 5 peers concurrently and pick the first responding peer as the proxy.
Figure 5.8: Degree of Separation for Various Network Sizes
34
In Figure 5.8, we plot the number of hops separating pairs of peers for networks of different sizes. We found that the UDP hole punching success rate for the Karkinos data set at 95% is similar to that for Tankie MAX, as shown in Table 5-1, while the TCP success rate is significantly lower at less than 60%. When nodes are separated by two hops, a proxy will be required. We see from these results that for UDP, a proxy can always be found when there are at least 4 nodes, while for TCP, a proxy can always be found when there are at least 7 nodes. Since the Karkinos data set is relatively small and has TCP hole punching success rates that are significantly lower than that for the larger Tankie MAX data set, it is plausible that TCP proxies might be easier to find in practice.
5.2.4 Proxy Performance
Figure 5.9: Round-trip latencies for direct and proxied UDP Connections
In Figure 5.9, we plot the round-trip latencies observed for different types of UDP connections for both OpenNAT and RakNet. On the whole, we see that direct and server proxied connections are comparable with peer-proxied connections, which are slower by about 20 to 30 ms. We believe that this is due to the extended queuing and processing delays in a peer proxy over a dedicated server.
35
Figure 5.10: Round-trip latencies for direct and proxied TCP connections
In Figure 5.10, we compare the measured round-trip latencies observed for direct and proxied TCP connections by sending a ping signal down the channel and measuring the time it takes to receive a response. Results are similar to that for UDP, though proxied TCP connections have significantly higher latencies. This is likely a result of the segmentation and buffering in our TCP proxy implementation for recovery purposes when a peer proxy leaves the network.
Figure 5.11: Throughput for direct and proxied UDP connections
36
Figure 5.12: Throughput for direct and proxied TCP connections
We also investigated how NAT proxies would affect the achieved throughput. In Figure 5.11 and Figure 5.12, we plot the measured throughputs for direct and proxied connections for UDP and TCP respectively. For OpenNAT, UDP throughput is estimated by sending as much data as possible within an interval of 2.5 s, while TCP throughput is estimated by sending 2 MB of data between end-hosts. The values of 2.5 s and 2MB were chosen because we found that these were the minimum values required to achieve good measurement accuracy after a calibration test. The graphs show that peerproxied connections are generally slower than direct and server-proxied connections, but still achieve reasonably high throughput. One interesting observation of both graphs is that there are 3 distinct points (~450Kps, ~650Kbps and ~1050Kbps) where there is a dramatic change in gradient. We found out that it coincides with the upload limit of broadband plans available in Singapore. Figure 5.11 also shows the UDP throughput for RakNet, which was measured differently because of its built-in flow control mechanism. We filled the transmit buffer with 400 KB of data and let RakNet control the actual sending rate. We see that the achieved throughput for RakNet is much lower than our implementation. The graph is also smoother as the send rates do not seem to reach the bandwidth limits of the users’ Internet connections.
37
Figure 5.13: Loss rates for UDP connections
The effectiveness of this flow control mechanism is apparent in Figure 5.13, which shows that RakNet achieves significantly lower loss rates for UDP packets as compared to our implementation. We also investigated the performance of our protocol in the case that a proxy fails and an alternative peer is required to restore the proxied connection. We found that in our implementation, a proxy connection can typically be restored within 6 s for UDP and within 18 s for TCP.
5.3 Preliminary Study on 3G Mobile Phone-to-Phone Connections To facilitate future implementation of the OpenNAT library on mobile platforms, we conducted a preliminary study on Singapore’s 3G mobile carrier networks to better understand their NAT infrastructure and the difficulties involved in mobile phone-to-phone connections. Figure 5.14 shows the components of our experimental setup, which consists of two laptops that are each connected to a mobile phone using a USB cable and the OpenNAT back-end system. We used the mobile phones to provide 3G Internet access to the laptops through a process known as tethering (Wikipedia, 2010). The test client that was embedded in Karkinos was modified to be used in this study. 38
Figure 5.14: Experimental Setup for 3G Mobile-to-Mobile
Our experiments were carried using six different SIM cards; two SIM cards from each mobile carrier – M1, Singtel and Starhub. After each experiment, the SIM cards were replaced and tested again until all combinations of the three carriers have been tested. The same laptops and mobile phones were used throughout the entire experiments. Mobile-One
Singtel Mobile
Starhub Mobile
Mobile-One Singtel Mobile Starhub Mobile Table 5-3: UDP and TCP Direct Connection
For the first test, we initiated a direct UDP and TCP connection between the mobile phones. The order of connections between the mobile phones is insignificant as the OpenNAT protocol requires both phones to attempt a connection to each other. Table 5-3 shows that direct connections for both UDP and TCP is possible when both phones are on the same mobile carrier network. This result is expected as the carrier network resembles the conditions of a local area network where applications can communicate directly with one another without the need of going through NAT (Figure 5.15).
39
Figure 5.15: Mobile Carrier Network
Mobile-One Mobile-One
Singtel Mobile
Starhub Mobile
– –
Singtel Mobile
–
Starhub Mobile Table 5-4: Cross-carrier UDP NAT Traversal
For the next test, we attempted to establish cross-carrier connection by performing NAT traversal using the OpenNAT library. Table 5-4 shows that direct cross-carrier UDP connections with Mobile-One network was possible with NAT traversal. Direct UDP connections between Singtel and Starhub networks were not possible and a proxy has to be used to relay data between the networks. Table 5-5 shows the results of our attempt to perform TCP NAT traversal. Direct TCP connections were possible between Singtel and Mobile-One networks with NAT traversal. However unlike the previous results, direct TCP connections between Starhub and MobileOne networks were not possible even with NAT traversal. At this point, we do not have a good explanation for this phenomenon and a more comprehensive study is required. The average latency of UDP connections observed for the whole experiment is about 175 ms and the difference in latency between same carrier-pair networks is about 25 ms. 40
Mobile-One
Singtel Mobile
Starhub Mobile
–
Mobile-One
–
Singtel Mobile
–
Starhub Mobile Table 5-5: Cross-carrier TCP NAT Traversal
5.4 Code Comparison between OpenNAT and JavaSE We created two chat applications in UDP and TCP to demonstrate the simplicity of developing universally connectable devices using the OpenNAT network library. Our implementation shows that there are minimal differences (highlighted in yellow) when using the OpenNAT library compared to the JavaSE 6.0 platform. To keep the examples neat, exceptions and error handling are excluded in the following code snippets. For the full source code, please refer to Appendix A. OpenNAT Library
JavaSE 6.0
// Register with OpenNAT System IdentityService.register("Qingwei.PC"); // Create Java DatagramSocket DatagramSocket socket; socket = new DatagramSocket(8888);
// Create OpenNAT DatagramSocket DatagramSocket socket; socket = new DatagramSocket(8888);
Table 5-6: UDP DatagramSocket Initialization Comparison
OpenNAT Library
JavaSE 6.0
SERVER
SERVER
// Register with OpenNAT System IdentityService.register("Qingwei.SERVER"); // Create OpenNAT ServerSocket ServerSocket serverSocket; serverSocket = new ServerSocket(8888);
// Create Java ServerSocket ServerSocket serverSocket; serverSocket = new ServerSocket(8888);
Socket clientSocket; clientSocket = serverSocket.accept();
Socket clientSocket; clientSocket = serverSocket.accept();
CLIENT
CLIENT
// Register with OpenNAT System IdentityService.register("Qingwei.MOBILE"); // Create OpenNAT Socket and connect to // server Socket clientSocket; clientSocket = new Socket("Qingwei.SERVER" ,8888);
// Create Java Socket and connect to // server Socket clientSocket; clientSocket = new Socket("192.168.." ,8888);
Table 5-7: TCP ServerSocket / Socket Initialization Comparison
41
Table 5-6 and Table 5-7 show a slight difference between OpenNAT and JavaSE socket initialization. For the OpenNAT library, an additional step is required to register the application with the OpenNAT system. After successful registration, the application will be contactable by other clients. OpenNAT Library
JavaSE 6.0 // Send message to peer send(InetAddress address, String msg) {
// Send message to peer send(Node node, String msg) { // Create Datagram Packet DatagramPacket p = new DatagramPacket( msg.getBytes(), node, 8888); // Send message socket.send(p); }
// Create Datagram Packet DatagramPacket p = new DatagramPacket( msg.getBytes(), address, 8888); // Send message socket.send(p); }
Table 5-8: Sending DatagramPacket Comparison
OpenNAT Library
JavaSE 6.0
CLIENT and SERVER
CLIENT and SERVER
// Write to OutputStream send(String msg) { socket.getOutputStream().write(..); socket.flush(); }
// Write to OutputStream send(String msg) { socket.getOutputStream().write(..); socket.flush(); }
Table 5-9: TCP OutputStream Comparison
When sending a DatagramPacket, the only difference between OpenNAT and JavaSE is the addressing of the destination (Table 5-8). The OpenNAT library uses a Node class to identify clients when creating the DatagramPacket. The Node class can be obtained by using the IdentityService.lookup(String
identity)
method.
For
the
TCP
chat
example,
the
implementation is the same for both libraries (Table 5-9). OpenNAT Library
JavaSE 6.0 // Receive messages receive() {
// Receive messages receive() { // Create buffer byte[] buf = new byte[1024];
// Create buffer byte[] buf = new byte[1024];
// DatagramPacket to receive message DatagramPacket p = new DatagramPacket( buf, buf.length); // Listen for message datagramSocket.receive(p);
// DatagramPacket to receive message DatagramPacket p = new DatagramPacket( buf, buf.length); // Listen for message datagramSocket.receive(p); }
}
Table 5-10: Receiving DatagramPacket Comparison
42
OpenNAT Library
JavaSE 6.0
CLIENT and SERVER
CLIENT and SERVER
// Read from InputStream receive() { socket.getInputStream().read(..); }
// Read from InputStream receive() { socket.getInputStream().read(..); }
Table 5-11: TCP InputStream Comparison
Lastly, there are no differences between the two libraries when receiving a DatagramPacket or using TCP InputStream (Table 5-10 and Table 5-11)
43
Chapter 6
6. Conclusion Network Address Translators have been proven to work effectively in practice and has become a standard, indispensable feature in today’s routers. As a result, any p2p applications today will almost certainly require NAT traversal in order to establish end-to-end connectivity. Therefore, we see the opportunity to provide developers with a simple platform to easily deal with such connectivity issues. Our first contribution would be the OpenNAT platform. The OpenNAT platform enables developers to lower development cost and improve the overall quality of their network applications. They will be able to develop applications that are universally connectable without having to deal with the complexity of setting up and maintaining additional servers to support NAT traversal in their applications. This will allow them to minimize extended server support and move these resources into other projects. End-users on the other hand, would also benefit from a more seamless and enjoyable experience as they no longer have to fuss over router and firewall configurations when using developers’ applications. They also would not need to remember IP addresses or determine their external public IP address when they want to host a server in a private network. In this project, we developed the OpenNAT system consisting of NAT Traversal Service with proxy support, Identity Resolution and Management Service and a client network library in JavaSE. To aid us in our implementation of NAT traversal and proxy support, we conducted a measurement study to better understand hosts connectivity and NAT conditions in the Internet landscape. To the best of our knowledge, there has not been a comprehensive study on the use of peers as relays for NAT traversal and hence, this is our second contribution.
44
Our results showed that it is easy to find peers capable of acting as NAT proxies and that the performance achieved is comparable to that of server-based proxies. We also found that the simple scheme of broadcasting to five peers concurrently and picking the first responding peer is sufficient to achieve good performance. We also evaluated the performance of our library against RakNet, a commercial game network library with NAT traversal support. The hole punching success rates of both libraries are consistent with the findings of Guha and Francis in 2005 (S. Guha P. F., 2005). We also found that RakNet’s hole punch timeout settings may be too short as our results indicated that it may take longer than 10 s to successfully perform a UDP hole punch. We submitted a separate research paper detailing the results of our experiments and it was accepted for publication at the 8th International Workshop on Network and Systems Support 2009 (D. Seah, November 2009). Our third and last contribution would be the preliminary study on 3G mobile carrier networks. To facilitate future implementations of the OpenNAT library on mobile platforms, we conducted a preliminary study on Singapore’s 3G mobile carrier networks to better understand their NAT infrastructures and the difficulties involved in mobile phone-to-phone connections. Our results showed that direct connections for both UDP and TCP are possible when both phones are on the same mobile carrier network. We also found that NAT traversal and proxy support are required for cross-carrier UDP and TCP connections. Lastly, we compared the implementation of a simple chat application using the OpenNAT library with the standard Java SE 6.0 platform. We showed that developing network applications that are universally connectable is simple and similar to traditional implementations.
6.1 Future Work As mentioned in Chapter 1.3 on the scope of the project, the remaining features of the OpenNAT system such as Mobile Proxy Service and library support for other platforms will remain as future work. As the OpenNAT identity service is similar to domain name systems (DNS), a more comprehensive study should be conducted to learn more about the advantages and shortfalls of DNS and applying the knowledge in our implementation. In additional, it is 45
essential for us to have a clear understanding of the problems associated with mobile network connectivity in order for us to provide a complete mobile solution. An in-depth study of the mobile carriers’ network as well as battery consumption of network activity on mobile phones should be conducted in the future. Lastly, the implementation of our NAT proxy support and load balancing algorithm is somewhat naive and therefore further research and optimizations have to be made to improve proxy performance and scalability.
46
References A. Razeen, D. S. (2009). Retrieved from Karkinos: http://apps.facebook.com/karkinos_alpha/ B. Ford, P. S. (2005). Peer-to-peer communication across network address translators. in Proc. ATEC’05. Bernard Traversat, A. A. (2003). Project JXTA 2.0 Super-Peer Virtual Network. Sun Microsystems. Carolyn Duffy Marsan. (2008, 7 21). NATs necessary for IPv6, says IETF chair. Retrieved 3 16, 2010, from NetworkWorld: http://www.networkworld.com/news/2008/072109-nathousley-qna.html Chappell, D. (2007). Introducing Windows Communication Foundation. Microsoft. D. Seah, W. K. (November 2009). Peer NAT Proxies for Peer-to-Peer Games. Proceedings of NetGames ’09. Deering, S. (1998). RFC 2460, Internet Protocol, Version 6 (IPv6) Specification. Internet Engineering Task Force. FierceDeveloper. (2009, 3 16). Smartphone sales increase 13.9% in 2008. Retrieved 3 15, 2010, from http://www.fiercedeveloper.com/story/smartphone-sales-increase-13-92008/2009-03-16 Ford, B. (2008). UIA: A Global Connectivity Architecture for Mobile Personal Devices. PhD thesis, Massachusetts Institute of Technology. Gas Powered Games. (2009, 4 14). (Stardock Entertainment and Gas Powered Games) Retrieved 3 21, 2010, from Demigod: http://www.demigodthegame.com/ Gnuplot. (2010, 3). Retrieved from http://www.gnuplot.info/ H. X. Nguyen, D. R. (2008). Balanced relay allocation on heterogeneous unstructured overlays. in Proc. INFOCOM’08. International Organization for Standarization. (2008). ISO/IEC 29341-1, UPnP Device Architecture 1.0. Internet World Stats. (2009, 9 30). World Internet Users and Population Stats. Retrieved 3 2010, 15, from http://www.internetworldstats.com/stats.htm
47
J. Dowling, J. S. (2007). Improving ICE service selection in a p2p system using the gradient topology. in Proc. SASO’07. J. Rosenberg, H. S. (2002). RFC 3261, SIP: Session Initiation Protocol. Internet Engineering Task Force. J. Rosenberg, R. M. (2005, 2). Internet Draft: Traversal Using Relay NAT (TURN). Retrieved 4 4, 2010, from Internet Engineering Task Force: http://tools.ietf.org/id/draft-rosenbergmidcom-turn-04.txt J. Rosenberg, R. M. (2008). Session Traversal Utilities for NAT (STUN). RFC 5389. J. Yong, J. B. (2009, 4). Retrieved from Tankie MAX: http://apps.facebook.com/tankie_max Jenkins Software. (2010). RakNet - Multiplayer game network engine. Retrieved 3 2010, 19, from http://www.jenkinssoftware.com/ Klein, A. (2007). BIND 9 DNS Cache Poisoning. Trusteer. Laptoplogic.com. (2008, 12 26). Laptop Sales Exceed Desktop Sales Globally. Retrieved 3 2010, 15, from http://laptoplogic.com/news/laptop-sales-exceed-desktop-sales-globally-20319 Li, G. (2001). JXTA: a network programming environment. 5 (3). N. Asokan, V. N. (2005). Man-in-the-Middle in Tunnelled Authentication Protocols. Springer Berlin / Heidelberg. NetworkWorld. (2009, 3 27). IETF: No consensus on IPv6 NATs. Retrieved 3 24, 2010, from http://www.networkworld.com/news/2009/032709-ietf-ipv6-nats.html Patel, N. (2009, 11 30). Strategy Analytics. Retrieved 03 15, 2010, from Global handset data traffic forecast 2001–2013: http://www.strategyanalytics.com/default.aspx?mod=ReportAbstractViewer&a0=5180 PCWorld. (2008, 8 19). Study Shows Glacial Pace of IPv6 Adoption. Retrieved 3 16, 2010, from http://www.pcworld.com/businesscenter/article/150013/study_shows_glacial_pace_of_ipv6_ adoption.html Postel, J. (1994). RFC 1591, Domain Name System Structure and Delegation. Internet Engineering Task Force. Rachna Dhamija, J. D. (2006). Why Phishing Works. Human Factors in Computing Systems. ACM. S. Guha, N. D. (2006). An experimental study of the Skype peer-to-peer VoIP system. in Proc. IPTPS’06. 48
S. Guha, P. F. (2005). Characterization and measurement of TCP traversal through NATs and firewalls. in Proc. IMC’05. S. Guha, Y. T. (2004). NUTSS: A SIP-based approach to UDP and TCP network connectivity. Proceedings of SIGCOMM Workshops ’04 Future, Directions in Network Architecture.: ACM. S. Ren, L. G. (2006). ASAP: an AS-aware peer-relay protocol for high quality VoIP. in Proc. ICDCS’06. Saint-Andre, P. (2004). RFC 3920, Extensible Messaging and Presence Protocol (XMPP). Internet Engineering Task Force. W. Kho, S. A. (2008). Skype relay calls: Measurements and experiments. in Proc. INFOCOM’08. Wikipedia. (2010). Tethering. Retrieved 3 20, 2010, from http://en.wikipedia.org/wiki/Tethering
49
Appendix A – UDP Chat Application Example import import import import import
java.io.*; alf.adu.*; opennat.impl.datagram.*; opennat.client.*; opennat.*;
public class UDPChatClient { public static final int CHANNEL_ID = 8888; private DatagramSocket datagramSocket; public UDPChatClient() throws ChatException { // Register with OpenNAT Identity Resolution Service if (IdentityService.getInstance().register("Qingwei.MOBILE")) throw new ChatException("Unable to register with OpenNAT System."); // Create OpenNAT DatagramSocket and bind to Channel 8888. datagramSocket = new DatagramSocketImpl(CHANNEL_ID); createListenThread(); } // Send message to peer public void send(String identity, String msg) throws ChatException { // Lookup address of identity Node node = IdentityService.getInstance().lookup(identity); if (node == null) throw new ChatException("No such identity in network."); try { datagramSocket.send(new DatagramPacketImpl(msg.getBytes(), node, CHANNEL_ID)); } catch (IOException e) { e.printStackTrace(); } } // Listening thread public void createListenThread() { new Thread() { @Override public void run() { // Create DatagramPacket to receive message DatagramPacket packet = new DatagramPacketImpl(); while (true) { try { // Listen for message datagramSocket.receive(packet); // Print message System.out.println(new String(packet.getData())); } catch (IOException e) { e.printStackTrace(); } } } }.start(); } }
50
