1

MaidSafe-Routing: a novel overlay for decentralised development MaidSafe staff

Abstract—This paper introduces the novel MaidSafe-Routing DHT. MaidSafe-Routing is an open-source C++ implementation of DHT that improves on Kademlia by adding features to significantly improve the overall performance and accuracy of the DHT. The simulation and experiments shows that MaidSafeRouting can be adopted as a highly efficient communication component of any decentralised overlay application.

I. I NTRODUCTION DHTs are scalable, efficient, robust and self-organising routing overlays for Internet-wide P2P deployment. Chord [10], Pastry [9] , Kademlia [7] and CAN [8] are a number of well known DHT schemes. Kademlia, due to its unique features is the most outstanding DHT and is widely implemented [5], [6], [2] and employed by public networks, including BitTorrent [1], Kad [4] and Gnutella [3]. The MaidSafe-Routing DHT proposed and implemented by MaidSafe.net to address the increasing demand for a highly efficient communication platform, desired by emerging decentralised Internet era. MaidSafe-Routing, inspired by the novel Kademlia DHT, preserves most interesting features of Kademlia, including • • •

•

•

the binary tree structure, using XOR for denoting distance in address space, offering an efficient routing algorithm by populating the routing table with nodes in different parts of the address space (relative to own node id), populating the routing table with nodes in close proximity to provide a fast and accurate convergence to the target, and caching mechanism.

MaidSafe-Routing improves on Kademlia by •

• • •

• • •

•

maintaining the structure using live connections to the neighbouring nodes, encrypted communication, user anonymity, enhancing a nodes awareness of the nodes in its close proximity by introducing group matrix, reliable unicast and group communication, NAT traversal services, enabling a node to join the network in a variety of operation modes, offering APIs and call backs.

By adopting the above enhancements MaidSafe-Routing offers • • •

significantly lower communication delays, secure communication, immunity to most network attacks,

handling churns efficiently by reflecting the event to related nodes’ routing tables and transferring relevant data to appropriate nodes immediately, and • adopting MaidSafe routing as an ideal communication component of any P2P applications. The rest of the paper proceeds as follows. Section II describes connectivity in MaidSafe-Routing using live connections between nodes. In Section III two types of reliable communication offered by MaidSafe-Routing is discussed. Different modes of operation for a node, and features of the node in each are covered in Section IV. Section V explains a variety of routing tables a node may have in a MaidSafe-Routing node, followed by introducing group matrix in Section VI. Introduction of protocol rpcs; bootstrapping; NAT traversal; and MaidSafe-Routing APIs and callbacks are described in Sections VII, VIII, IX and X, respectively. Finally, the paper presents an analytical and experimental evaluation of the network in Section XI. •

II. M ANAGED CONNECTIONS . The efficiency of communication in a P2P network heavily depends on the size and accuracy of the routing table of nodes in the network. The routing table in Kademlia is populated according to the elegant concept of bucket. Each bucket stores only information about limited number of peers which are in a known range of distance from the node. Given there is one bucket for each bit in the node identifier and each bucket stores multiple entries, Kademlia should have enough number of nodes to offer an efficient look up. However, as in Kademlia a node’s leaving is not immediately reflected to the routing table of the holding peers, the routing table is not accurate, and that is why Kademlia relies on multiple parallel messages to nodes close to the target. Unlike Kademlia which stores only node id and endpoint of the peers, in MaidSafe-Routing each entry in the routing table is associated with a live RUDP (Reliable UDP) connection provided by MaidSafe-RUDP. RUDP implements psuedoconnections using UDP to achieve many of the benefits of a connection-based protocol like TCP, but crucially allows NAT traversal where TCP cannot. Populating the routing table with live connections in MaidSafe-Routing significantly improves the accuracy of the routing table, since any join or leave is immediately noticed and reflected to the appropriate nodes routing table. Hence, improving routing efficiency. III. R ELIABLE UNICAST AND GROUP MESSAGING MaidSafe-Routing through the services at MaidSafe-RUDP offers encrypted messages in all communications. MaidSafe-

2

Routing employs symmetric encryption, in which the key is exchanged using an asymmetrically encrypted communication at the start of the connection between a pair nodes. Hence, achieving the security of asymmetric encryption at the cost of symmetric encryption. Similar to Kademlia and most other DHTs, a message typically traverses a number of nodes before reaching the destination. An intermediate node in MaidSafe-Routing does not pass the endpoint information of its previous node to the next node. Therefore, it is not possible to trace back the origin of a message. This feature offers the anonymity which might be crucial to some users. MaidSafe-Routing offers two types of communication, namely, unicast and group. Both types of communication offer reliability through acknowledgment messages. a) Unicast messaging: In unicast messaging the destination is a known node in the network and is the final recipient of the message. At each intermediate hop, an acknowledgment is sent to the previous hop, after successfully sending message to the next hop. The destination hop, sends acknowledgment back to its previous hop on receiving the message. An intermediate node may re-sends the message to the appropriate node multiple times if no acknowledgment is received after a deadline. b) Group messaging: A group is comprised of a number of nodes which are closest to a given target. The number of group members is defined as a configuration setting. In group communication, the destinations of a message is a group of of nodes whose node ids are closest to the target. A node having the target as its node id is not considered as part of the group. Delivering a group message is performed in two stages. In the first stage the message is delivered to the node with the closest id to the target. In the next stage, the closest node, using the local information in group matrix (will be covered in Section VI) finds the rest of group members and sends unicast message to each of them. In parallel to sending messages, the closest node, processes the message as a group member. The reliability in group communication is achieved similar to unicast communication up to the time when the message is received by the closest node. The closest node send acknowledgement to its previous node after receiving the acknowledgement from the majority of the group members. IV. M ODES OF

OPERATION

A node in MaidSafe-Routing may join the network in a client or non-client mode.

B. Client In contrast to non-client nodes, a client node does not take part in the structure and connectivity of the network. Client nodes are light weight routing nodes using minimal network resources. A client node gain access to the entire network by establishing connection to a fixed number of non-client nodes closest to its own id. A client node is able to: • send requests to any non-client nodes, • send requests to clients having similar ids as the client node itself, and • receive incoming requests only from connected non-client nodes. In the rest of the paper we refer to non-client node as node. V. ROUTING TABLE In MaidSafe-Routing, as a P2P network, each non-client node has the same capability as any other non-client nodes. Therefore, the operation of network depends on the contribution of every non-client node in maintaining the structure and connectivity of the network. In fact, the reliable operation of the network to deliver messages, and accuracy of decisions made at higher levels to a large extent depends on how nodes may connect to each other. Enabling communication between each pair of nodes using a distributed algorithm where each node efficiently takes part in routing decisions requires each node to have partial knowledge about a number of other nodes in the network. In such a scheme each intermediate node receiving a message, using the local information should be able to send the message closer to the destination. Given, that the routing table has limited resources, utilising the available resources in such a way to offer network connectivity and efficient routing decision makings could be a challenge in reality. Populating the routing table with only close nodes results in efficient routing decisions, when message is in close proximity of the target. However, it may result in, i) message traversing many hops, and ii) network segmentation. On the other hand, lack of close nodes around a node may lead to inefficient routing decision when the message approaches the target. Therefore, efficient and reliable routing may be realised only if a combination of close and far nodes exist in the routing table of each node. The client nodes do not participate in routing decision makings, therefore, the routing table in client nodes is different from ones in a non-client node. The rest of this sections describes routing tables in non-client and client nodes in MaidSafe-Routing. A. Routing table in non-client nodes

A. Non-Client A non-client node is a full routing node that contributes to the structure, connectivity and maintenance of the network. Non-client nodes are part of DHT and are active in routing decision makings. A non-client node is able to: • • •

send requests to non-client nodes, send requests to only connected client nodes, and receive incoming requests from any node in network.

Depending on the type of the peers, an entry associated with each MaidSafe-RUDP connection in a non-client node may be reflected in to, routing table, client routing table or non-routing table. 1) Routing table: The entries in routing table of a MaidSafe-Routing node are used for making decision on transmitting a message to the next node. A routing table may have up to 64 entries, and an entry in the routing table may be:

3

• • •

one of the closest nodes, a random nodes in the network, or a node which finds the current node as one of their closest nodes, while the current node does not consider it close enough.

The rest of this section further describes the structure the routing table in MaidSafe-Routing. a) Connection to closest nodes: Establishing connections to the 8 closest nodes is vital in reliable operation of a node in MaidSafe-Routing, and the library offers efficient algorithms that enables each node to find and connect to its 8 closest nodes. The distance between a pair of nodes is calculated by XORing the node ids of the pair. And, obviously, the 8 closest nodes chosen for a node have the least distance to the node compared to any other nodes in the network. By Cni , we denote a group of 8 closest nodes to node ni , i.e. ∀nj ∈ N etwok ∧ nj ∈ / Cni , ∄nx ∈ Cni | nj ⊕ ni < nx ⊕ ni where Cni denotes the list of nodeni’s closest nodes, and ⊕ is XOR operator. b) Connection to random nodes: To enable connectivity between each pair of nodes, in addition to connection to the closest nodes, MaidSafe-Routing offers services that each node connects to random nodes in the network to offer uniform access to different parts of the network. Actually, these connection use a significant part of the routing table at each node. Similar to Kademlia, MaidSafe-Routing adopts the concept of bucket to arrange neighbouring peers and to offer access to different parts of the network. c) Connection to nodes with unidirectional interest: Due to the binary tree structure of MaidSafe-Routing, nodes with close ids reside in close proximity of each other and have similar view about their closeness to each other. There are, however, circumstances where one node may find another node as one of its closest nodes, but, the peer appropriately does not consider another one as one of its closest nodes. This is simply due to the presence of a few more close nodes to one node. As proper operation of the network depends on each node to be connected to its 8 closest nodes, in MaidSafeRouting, 16 connections are allocated for the nodes which have unidirectional interest to the current node. We denote unidirectional interest connections to node ni by CUni , which can be expressed as: / Cni } CUni = {nj | ni ∈ Cnj ∧ nj ∈ 2) Client routing table: The client nodes do not take part in routing decisions, therefore, their corresponding entries are stored in client routing table of a non-client node. A client routing table may have up to 64 nodes. 3) Non-routing table: The non-routing table contains information associated to the connections to the non-client nodes which are not considered suitable to be involved in routing decision makings. Such nodes include those which have conducted malicious activities, or the non-client nodes that exhibit poor performance.

B. Routing table in a client node A client node in MaidSafe-Routing network may connect only to non-client nodes. A client node requires connections to 8 closest (non-client) nodes to be considered healthy enough to be allowed to start any communication. VI. G ROUP M ATRIX Churn is a normal behaviour of any P2P network. Achieving data availability in such a dynamic network requires that data to be stored on a number of nodes and to be replicated to new nodes once a storing node leaves the network. Similar to the Kademlia DHT, in MaidSafe-Routing key-values are stored to multiple nodes which are closest to the key. According to MaidSafe-Routing terminology these multiple nodes are called a group. It is crucially important that local information at the storing nodes to be accurate enough to ensure that no other node in the network is closer to the key than the storing group. Obviously, such an accurate decision requires each node in storing group to have knowledge about the nodes in their close proximity. The information stored in routing table is enough in the majority of cases to enable correct decision makings. However, it might be found insufficient to make accurate decisions when two nodes are having different views regarding closeness to each other or another node. To handle these situations, the routing table is equipped with a group matrix. The idea behind group matrix is to increase a node’s awareness to its close proximity, without the need for scarce connections. This is realised by making each node partially aware of nodes in the routing table of its closest neighbours and the nodes which have unidirectional interest to the node, we call these nodes total closest nodes and denote it by CTni for node ni . In other words, CTni = Cni ∪ CUni For each node nj in total closest nodes of ni , the group matrix of ni has an associated row. We denote the row for nj by GRni,j . The row is comprised of nodenj plus total closest nodes to nj , i.e. GRni,j = nj ∪ CTnj . Therefore the matrix of a node ni , denoted by GRni may be expressed as Gni = {∪GRni,x | nx ∈ CTni }. To maintain the local information at each node up to date, any modifications to a node’s total closest peers is immediately reported to all nodes in total closest nodes. Moreover, after each modification the group matrix is investigated to prune unnecessary rows. A row associated to nodenj in group matrix of node ni is removed, if no nodes finds the other as close to the other one, in other word, node ni removes GRni,j from its group matrix only if, / Cni . ni ∈ / Cnj ∧ nj ∈

4

VII. P ROTOCOL RPC S To maintain the structure, connectivity and accuracy of the network, the MaidSafe-Routing offers a number of RPCs, which are described in this section. a) Find node: Find node RPC retrieves the node id of a node whose id is closest to a given target. Unlike the Kademlia DHT, Find node RPC in MaidSafe-Routing is not iterative. Find node RPC can return a number of closest nodes to the target. Retrieving more than one close nodes to a target does not involve more network activity as group matrix of closest nodes using its group matrix can provide the desired nodes. b) Connect: MaidSafe-Routing allows connection between i) two non-client nodes or, ii) between a client and a non-client node. The connection between two nodes in MaidSafe-Routing requires exchanging more than one RPC. In the first round a connection request is sent by a node ni to a non-client node nj . If the destination node nj replies with success, two nodes attempt to create a connection using MaidSafe-RUDP. Once the connection is established nodes inform each other by another message showing the success. The success message sent form the requested node nj is populated with a list of closest nodes to the requester node ni using its local information. When the success message is received by the initial requester node ni it authenticates and adds node nj to its routing table. Moreover, it investigates the list of nodes sent by nj and attempts to connect to any one which must belong to Cni . Node ni , on receiving success message from nj , authenticates and adds node ni to its routing table if node ni is a non-client node. Otherwise, if node ni is a client node, it is added to the client routing table without any authentication. c) Remove: MaidSafe-Routing strives to make an efficient use of the limited available connections to have the most appropriate nodes in routing table. Once a MaidSafe-Routing node realises that a connection to a peer is not required any more, it sends a Remove RPC to the peer. The peer investigate its routing table and drops the connection if the requester is not one of its required closest node. d) Group update: Group update RPC is used to send group matrix updates to the relevant nodes. The group update is sent once there is a change in the total closest node list, CTni of a node ni . e) Ping: Is used to know about presence of a particular node in the network. VIII. B OOTSTRAPPING As mentioned in Section IV a MaidSafe-Routing node can be client or non-client. Due to the role of non-client nodes in structure and connectivity of the network, MaidSafe-Routing requires them to be authenticated. MaidSafe-Routing relies on Public Key Infrastructure to achieve the authentication. f) Client join: To join the network, a client which already has a node id, requires a bootstrap node. The client uses the bootstrap as a proxy to send a Find Node (with client node id as target) request to the network. The request is replied by the network through the bootstrap node. Obviously, the reply is

the closest node to the target, which client attempts to connect to. As described earlier as a part of Connect RPC the client is provided with a list of close nodes to which it connects. Once the number of connections reaches 8 the client is considered as joined. g) Non-client join:: A node may join the network as a non-client one only if it has its public key stored in the network. To accomplish this, a non-client node joins the network in two stages. In the first stage the node joins as a client, followed by storing its public key to the network. Once the public key is stored, in the second stage, the node re-joins as a non-client node which can be authenticated..

IX. NAT T RAVERSAL An objective of MaidSafe-Routing has been enabling communication between each pair of nodes in the network, regardless of their network configuration settings. MaidSafe-Routing in combination with MaidSafe-RUDP performs hole punching to enable direct connection between each pair of nodes. Hole punching is achievable as long as both nodes are not behind symmetric routers. If both nodes are behind symmetric routers, MaidSafe-Routing enables communication between two nodes by choosing a third node acting as proxy between the two nodes behind symmetric routers.

X. API S

AND

C ALLBACKS

Offering an efficient platform to exchange messages between peers makes routing an ideal communication component of any P2P system. To allow simple and loosely coupled utilisation of the routing components, MaidSafe-Routing offers a number of callback functions to the host components. A few of these callbacks are mandatory and must be implemented by the host application. While, other callbacks are optional for a host application. Followings are a list of callback functions provided by MaidSafe-Routing: h) MessageReceivedFunctor: The most basic functionality a host may expect from the communication component is to abstract away the underlying communication complexities and to be informed of the messages which are destined for the host. MessageReceivedFunctor functor is called when the host node is the one which is responsible for handling the message. i) NetworkStatusFunctor: Is called when a new connection is added or removed from the routing table. It shows the health in terms of the number of available connections. j) MatrixChangedFunctor: Any churn event, resulting in change to node’s group matrix triggers a call to this functor. k) RequestPublicKeyFunctor: As a part of the connection process to a non-client peer, routing needs to be provided with the public key of the peer. This is achieved by PKI infrastructure. The callback provides another callback (GivePublicKeyFunctor) which should be called with the valid public key of the connecting non-client peer l) CacheFunctor: Are called to look for cached data or store data in cache in an intermediate node.

5

m) NewBootstrapEndpointFunctor: Is called whenever a routing node connects to a peer whose endpoints are capable of bootstrapping a node. Since the network is very dynamic, this is important information for reconnecting to the routing network. A list of these endpoints must be supplied to routing to reconnect to the network. [APIs to be discussed] XI. P ERFORMANCE

AND ACCURACY EVALUATION

XII. C ONCLUSION R EFERENCES [1] [2] [3] [4] [5] [6] [7]

Bittorrent,: http://en.wikipedia.org/wiki/bittorrent. Entangled: http://entangled.sourceforge.net/. Gnutella: http://en.wikipedia.org/wiki/gnutella. Kad network: http://en.wikipedia.org/wiki/kadnetwork. Kashmir: http://kashmir.sourceforge.net. Vuze: http://wiki.vuze.com/index.php/dht. Petar Maymounkov and David Mazières. Kademlia: A peer-to-peer information system based on the xor metric. pages 53–65, 2002. [8] Sylvia Ratnasamy, Paul Francis, Mark Handley, Richard Karp, and Scott Shenker. A scalable content-addressable network. In IN PROC. ACM SIGCOMM 2001, pages 161–172, 2001. [9] Antony Rowstron and Peter Druschel. Pastry: Scalable, decentralized object location, and routing for large-scale peer-to-peer systems. IN: MIDDLEWARE, pages 329–350, 2001. [10] Ion Stoica, Robert Morris, David Liben-Nowell, David R. Karger, M. Frans Kaashoek, Frank Dabek, and Hari Balakrishnan. Chord: A scalable peer-to-peer lookup protocol for internet applications. In ACM SIGCOMM, pages 149–160, 2001.