Real-Time Communications in Large-Scale Wireless Networks

There is an emerging need for realizing real-time quality of service (QoS) over multihop wireless communications in large-scale wireless networks. The applications can include wireless mesh infrastructure for broadband Internet access supporting multimedia services, visual sensor networks for surveillance, and disaster-relief networks. However, a number of challenges still exist as revealed by recent works, where the dataﬂow QoS performance such as throughput and end-to-end delay can degrade fast with the number of wireless hops. We propose to use large-scale cognitive networking methods to resolve the wireless multihop challenges. By the cognitive-networking concept, data packets travel along opportunistically available paths in the network with opportunistically available spectrum in every hop. Reliable end-to-end communications can be achieved for real-time services, where we show that (1) dataﬂow throughput can be independent of any number of wireless hops, (2) end-to-end delay and delay variance increase linearly with the number of wireless hops, and (3) delay variance decreases to zero with higher network density. These results are supported by analysis, simulations, and experiments.


INTRODUCTION
Large-scale wireless networks, for example, mobile ad hoc networks, wireless mesh networks, as well as wireless sensor networks, have been receiving significant attention in the past few years.Recently, there has been growing interest in realizing real-time quality of service (QoS) over multihop wireless transmissions in large-scale wireless networks.For example, the driving commercial applications include wireless mesh infrastructure for broadband Internet access [1], supporting Voice over Internet Protocols (VoIP) or Multimedia over IP, and wireless sensor network with real-time event detection/reporting, especially in video surveillance, that is, visual sensor networks [2].
Compared to the classical many-to-one (starlike) networks, such as cellular phone networks and WiFi-(IEEE 802.11) [3] enabled wireless local area networks (WLANs), a wireless mesh-based infrastructure for broadband access has the advantage of much lower capital and operational costs and can achieve higher service coverage.As one example, municipal WiFi mesh networks are rolling out in a number of major cities, providing wireless web surfing and email services to general public.Considering the popularity of Internet voice and video, further developments for supporting VoIP within the wireless mesh infrastructure are highly demanded.On the other hand, visual sensor networks appear to be one of the killer applications for wireless sensor networks, since a lot of major cities have installed publicadministrated surveillance cameras for various purposes.As of now, those cameras are normally attached by cables.By adopting wireless connections, there will be significant cost reduction in the installation, which can enable much denser deployment.Moreover, related applications also include the disaster-relief wireless networks, with ad hoc formation, where the real-time information delivery can be of life and death importance.
However, many challenges still exist in realizing real-time QoS over multihop wireless networks.It has been well known that dataflow throughput, end-to-end delay, and delay jitter can degrade fast when the number of wireless hops increases [4,5].The limitation can be primarily introduced by dynamic network-resource availabilities including both spectrum bandwidth and wireless nodes/radios.Specifically, random spectrum availability is based on wireless fading International Journal of Digital Multimedia Broadcasting and interferences prevailing in unlicensed bands (e.g., ISM bands); and random radio availability is due to dynamic traffic load (congestions) and other factors such as radio failure.In traditional wireless networking, the media access control (MAC) layer sets up "logic wired links" over wireless media, which assume predetermined spectrum availability.On top of it, the network layer runs ad hoc routing protocols which assume predetermined radio availability and network topology.It is difficult to have effective QoS models in the traditional network protocol stack [6], when both spectrum bandwidth and radio availabilities cannot be predetermined in large-scale wireless networks.
In order to resolve the wireless multihop limitation, there have been proposals to install multiple radios in one single wireless node [4] and design network routing protocols based on wireless link status, as well as other combined metrics [5].Although evidence shows that improvements could be obtained, these approaches do not address realtime QoS.Moreover, existing approaches can generally suffer from the lack of scalability.Typically, the complexity of existing protocols can grow fast with the number of wireless nodes neighboring each other (higher network density), since there are a large number of links to be managed and/or tracked on every node.Other research work that investigates real-time QoS in wireless networks could be based on inappropriate assumptions for large-scale wireless networks, such as the assumption on fixed link bandwidth [7].
In this paper, we apply the methods of large-scale cognitive networking [8] to resolve the wireless multihop limitation.The concept of cognitive networking opportunistically utilizes network resources including both spectrum bandwidth and wireless node/radio availability to realize reliable communications in large-scale wireless networks.The adopted network architecture is the Embedded Wireless Interconnect (EWI) [8][9][10][11], where wireless linkage is redefined introducing abstract wireless links.By definition, abstract wireless links can be arbitrary functional abstractions of mutual cooperations among proximity wireless nodes, where the categories can include broadcast, unicast, multicast/anycast, and data aggregation, among others.Therefore, wireless link modules are designed as the units for realizing different abstract wireless links at individual wireless nodes.The architecture of EWI is then developed upon two layers, where the bottom wireless link layer supplies a library of wireless link modules; and the upper system layer can organize the modules for achieving effective application programming interface (API).
Conforming to the cognitive networking concept, both the operating spectrum and the participating nodes of an abstract wireless link is opportunistically decided based on their instantaneous availabilities.By the unicast module design in the cognitive networking, data packets can take opportunistically available paths in the network from source to destination, with opportunistically available spectrum in every hop.In comparison, dynamic changes in spectrum and radio availability can create bottlenecks on the routing path of traditional wireless networking, where such bottlenecks are all resolved by the proposed methods.Therefore, reliable end-to-end communications can be achieved, by the opportunistic utilization of network resources.The presented unicast module design is also implemented by cognitiveradio prototyping [12].In the performance analysis, realtime QoS metrics including throughput, expected end-toend delay, and delay variance, are studied.The following propositions are obtained.
(i) All the investigated real-time QoS metrics improve with larger network scale, and the complexity on individual nodes remains constant independent of any network scale.It is also shown that the performance dynamics, as indicated by end-to-end delay variance, can diminish to zero with higher network density.
(ii) Reliable communications can be achieved over a large number of wireless hops.In principle, the throughput can be independent of the source-todestination distance; while the expected end-to-end delay, and the delay variance, increase linearly with the source-to-destination distance.
(iii) Considering the resource constraints, such as energy consumption or network capacity, we identify that radio transmitting power can be an effective control knob for traffic prioritization in dealing with the tradeoff between achievable QoS performance and resource consumptions.
Therefore, the two major contributions of the paper are the following: (1) the first design & implementation of real-time communications for large-scale wireless networks; (2) the QoS performance analysis of the designed realtime unicast module.In what follows, related works are surveyed in Section 2; the unicast wireless link module design and implementation are described in Section 3; the performance analysis of real-time QoS metrics is presented in Section 4; the simulation and experiment results are reported in Section 5 and 6, respectively; the conclusion is given in Section 7.

RELATED WORK
Related works in cognitive networking and the EWI network architecture have previously appeared in [1,[8][9][10][11][12].As a pilot architectural reference model, EWI was first introduced in the application-specific studies in wireless sensor networks [10,11].In [9], recent research works in the cross-layer design of wireless sensor networks are surveyed, where it is suggested that EWI can be an unified design architecture.The concept of large-scale cognitive wireless networks has been presented in [8], where EWI was adopted as the network architecture.In [1,12], the EWI is further utilized to construct a ubiquitous wireless mesh infrastructure for broadband Internet access.
The unicast design can also be related to existing research works in opportunistic routing [13][14][15][16][17].In principle, opportunistic routing schemes deal with the codesign of routing and MAC protocols.However, previous research works usually deal with single-hop performance metrics, while the contribution of this paper deals with end-to-end QoS for real-time communications.Moreover, contributions are made in large-scale cognitive networking, with a report of experiment results.
In [13], the protocol selects the next hop relay node by a slotted ACK (acknowledge) mechanism.Having successfully received a data packet, the node calculates a priority level, which is inversely proportional to the distance between the node and the packet destination.The node with the highest priority is then selected distributively as the next hop relay node.In [14,15], a technique named GeRaF was proposed, where the node forwarding region is divided into smaller sections with different priorities, as decided by the forwarding distances.The performance analysis of GeRaF included the single-hop energy consumption and latency tradeoff, as well as the multihop count number.The study in [16] investigated the architectural aspects of opportunistic routing, where the performance of the component design is analyzed based on single-hop metrics and light-traffic scenario.In [17], the authors discussed the opportunistic relay node selection in a two-hop transmission scenario.The relay-node priority criterion is based on the gain of sourceto-relay channels and relay-to-destination channels.

MODULE DESIGN AND IMPLEMENTATION
A unicast wireless link is an abstraction of the proximity wireless nodes cooperation for unicasting.Since unicasting usually involves the information delivery from source to destination over multiple hops, the unicast module design decouples the end-to-end network behavior into proximity wireless nodes cooperation.In the following, some elements of the module design and implementation are presented.The module state-diagram is then shown.

Network address
Network address is related to the context, subject to a "cost of delivery" criteria.Let d and s denote the network addresses of the destination node and the source node, respectively.Given the destination d of a data packet at one wireless node n, a local parameter c n,d is assumed obtainable, which indicates the approximate or average cost of delivering the packet to the destination from the node n independent of dynamic changes.In large-scale wireless networks, the cost of delivery c n,d is usually a function of the distance from one local node n to the destination d.
In location-centric networks (e.g., [10]), where wireless nodes are aware of their own locations, for example, by global position systems (GPS) or triangulation estimations, the network address of one node n is solely decided by the node n location coordinates L n .Given the destination coordinates L d of a packet, c n,d = |L n − L d | can be readily calculated in geometry, defined as the specific distance.
In data-centric networks (e.g., [18]), the network address can be decided by the application specific data.The cost of delivery c n,d is the application data gradient, which can be assumed as a monotonically increasing function of the distance between n and d.
In a data-collecting or fusion network, for example, wireless sensor networks, the sink (data collector) can broadcast a number of identity advertisement packets, which is thereon flooded in the network, by broadcasting.Every node can count the average smallest number of hops from the sink, on receiving the advertisement packets.The count number can be used as c n,d for one node n.A similar approach appeared in [19] yet in a more general scenario, where the logic address, that is, a vector of the estimated distance or hop number to a group of anchors, is maintained for every wireless node.New nodes joining in the network can estimate their own logic address by acquiring the address of neighboring nodes.
Although other types of address can be applicable, we use the location address in this paper for analysis, simulations, and experiments.

Radio implementation
The terminology of cognitive radio was first suggested in [20] as an ideal-omnipotent radio which can take all the parameters into consideration for user-centric communications.Cognitive radio was later reviewed in [21,22] as the radio with dynamic spectrum access.In [8], two propositions are further suggested for the cognitive radio for large-scale wireless networks.
(i) The radio can opportunistically sense the spectrum resource, so that the selected spectrum usage will not be interfering with other on-going wireless communications.
(ii) The radio can opportunistically poll one or more other proximity radios onto the selected spectrum, so as to realize certain types of local cooperations.
The above two propositions can extend the concept of pure cognitive radio to cognitive networks, which implement both dynamic spectrum access and dynamic radio access.In the network architecture EWI, a wireless node can initiate an abstract wireless link, that is, certain types of local cooperations among a number of proximity wireless nodes.Hence, both the set of wireless nodes and the operating spectrum are opportunistically decided for an abstract wireless link.Further, the initiated link will not be interfering with other wireless communications, in accordance with the first cognitive proposition.
In the test-bed development of the paper, we prototyped an experimental tone-based cognitive radio [8,12].Specifically, the radio can access a group of predetermined data channels, where every data channel is also associated with two distinctive frequency tones, that is, one sensing tone and one polling tone.The sensing and polling tones are at distinctive frequency different from the data channel.The radio hardware is therefore composed of two transceivers, which are the tone transceiver and the data transceiver, respectively.On initiating an abstract wireless link, the radio senses for an available channel, with the vacant data channel and the vacant sensing/polling tones.It then broadcasts the polling tone associated to the selected channel, to poll its surrounding nodes.On detecting the rising edge of the polling tone, the surrounding nodes can decide to join in the initiated abstract wireless link based on their autonomous availability.On joining in an abstract wireless link, the radio of the surrounding nodes also broadcasts the sensing tone.As such, both sensing and polling tones protect wireless link modules from spectrum interferences.We note that the use of frequency (busy) tones has also appeared in numerous works of MAC protocols [23,24].However, here we have the first implementation over multiple channels, and such implementation is in the context of large-scale cognitive wireless networks.
The unicast wireless link module is programmed on the cognitive radio prototype (shown in Figure 1).The experiment hardware is a stack of two radio boards (TELOSB [25]).The radio board is composed of one IEEE 802.15.4 [26] compatible transceiver, Chipcon CC2420, one TI MSP430 Microcontroller, and the program interface.The top radio board is utilized as the tone transceiver, while the bottom radio board is utilized as the data transceiver.The two are wired up by the digital interface shown in Figure 1.Three independent data channels in the 2.4 GHz band are allotted to the radio platform, while the associated tones are also in the band of 2.4 GHz.The unicast module is programmed in the firmware of TI MSP430.

Packet relaying process
Under an initiated wireless link, the packet relaying process picks one available relay node and sends the data information from the source or a previous relay node to the new relay node.
Four types of packets are used in the packet relaying process, which are RTS, CTS, DATA, and ACK, respectively.RTS, CTS, and ACK are control packets, while DATA is the data information unit.Let node n p denote the relay node at the hop p, the next hop relay node n p+1 is found by the following procedure, which incurs local wireless nodes cooperation in proximity, that is, a unicast wireless link.
First, the node n p broadcasts a RTS, including the module type (unicast), the destination address d, and the self-address n p .Upon receiving the RTS at one node n, if the condition c n,d < c np,d is satisfied, the node n initializes a timer, with the timeout period as which is determined by the specific physical radio technology, for example, where C 0 is a constant, and n / = d.SIFS is the smallest (minimal) inter frame space (delay constant), which is composed of the module processing time and the transceiver RX/TX switch time.In our current test-bed implementation, T d (n p , n, d) is also slotted according to the minimal carriersensing time.Note that the condition c n,d < c np,d is enforced, so that the cost of delivery to the destination is always descending which prevents any loop.
The node n then backs off and monitors the data channel for the period T d (n p , n, d).If the data channel is free during that period, the node n replies with a CTS, declaring itself as the next hop relay node n p+1 .Otherwise, the node n quits the unicast module.As such, the node with the smallest cost c n,d should be elected as the next hop relay node n p+1 .After having received the CTS, the node n p transmits the DATA packet, and after receiving the DATA packet, the node n p+1 replies an ACK, completing the round of relaying.In the case that more than one node send out their CTS simultaneously, the transmissions will collide on some specific bits, for example, the node address, where the CTS packets are different.Since this collision can be detected by certain mechanism, such as cyclic redundancy check (CRC), at n p , the node n p can use an appended procedure to differentiate one node as n p+1 , either before or after the DATA transmission.
The described mechanism is illustrated in Figure 2. In the last hop, the destination d is always assigned with the minimal delay, that is, T d (n p , d, d) = SIFS.

Module state diagram
The state diagram of a single unicast module in the wireless link layer is shown in Figure 3, which can give a summary of the implementation.The definitions of the states and the associated transferring branches can be self-explainable, in accordance with previous descriptions.
The wireless link layer stays in the IDLE state, when no module is involved.It can initiate a unicast module, for example, on receiving the command from the system layer, so as to send certain amount of information (i.e., contained in one DATA packet) to a specified destination.In the IDLE state, the wireless link layer also monitors the set of polling tones, which are associated to the predetermined group of data channels.On detecting the rising edge of one polling tone, it sends one module request to the system layer.Upon approval, the wireless link layer listens to the RTS packet on the data channel, which is associated with the detected polling tone.Otherwise, it goes back to the IDLE state.Furthermore, after having received one DATA packet, the unicast module automatically relays it, if the current node is not the destination.Table 1 shows an example set of primitive functions on the interface between the system layer and the wireless link layer, as related to the unicast module.The parameter Priority in Table 1 can be used for specifying the traffic class, which will be explored further in Section 4.5.3.

Discussion
By opportunistically dealing with network resources including both spectrum bandwidth and radio availability, data packets take opportunistically available paths with opportunistically available spectrum in every hop.Since the unicast module operates opportunistically in proximity wireless nodes, the end-to-end performance of real-time QoS, can be statistically assured, adding up the sequential proximity operations.This discussion can provide an intuitive explanation of the performance analysis results in Section 4. Specifically, random spectrum availability is handled by the cognitive radio implementation, where a source or relay node tries to find an available data channel for initiating the unicast wireless link.Important aspects of random node/radio availability are as follows: the node deployment, mobility, and traffic congestion.In general, it indicates that the node n p , that is, the pth hop relay, would be uncertain about the availability of the next hop relay node, before the RTS probing.The RTS/CTS exchange in the module design opportunistically finds the next hop relay, in a group of available candidates.
In traditional computer networks, dynamic traffic load is limited by network congestion control.The network layer drops overflowed packets by queueing management, which is then detected by the transport layer.Such methods have been known to incur problems in wireless networks [27].By the real-time unicast module, the paradigm of the classical "network queueing" could be transferred to "queueing in network."In an individual relay node, a DATA packet is forwarded automatically without lengthy buffering in one local queue.Sequential DATA packets nominally take opportunistically decided paths from the source to the destination, which are buffered in the network, instead of any predetermined nodes.
Therefore, it can be intuitive to conceive that the QoS performance metrics should improve with larger network scale, since the network density contributes to the diversity (or more radio resource) that can be opportunistically exploited.This agrees with other theoretical results, for example, in [28], where it is stated that summed network transport capacity should increase with network scale, on the order of Θ( √ N), where N is the number of nodes.
Moreover, in the proposed method, any node does not need to consider or know every other possible node as a candidate for the next hop.In the formation of abstract wireless linkage, the participating nodes are decided by their autonomous availability.The unicast module design also assures that there would only be one RTS response in most cases by the carrier-sensing mechanism.Therefore, the module complexity at individual nodes remains constant, independent of the network scale.

PERFORMANCE ANALYSIS OF REAL-TIME QoS
The performance analysis mathematically quantifies the major propositions in the paper.The results can also provide useful network planning guidance for deployments.

Objectives and approaches
In supporting real-time QoS, the metrics of throughput, expected end-to-end delay, and delay variance, are studied for the unicast wireless link module.Two types of resource consumptions are considered, which are network capacity and network energy consumption.Therefore, we centrally explore the following three questions.(1) How do the QoS metrics change with network scale?(2) How do the QoS metrics change with the source-to-destination distance?(3) How to control the resource allocation, so as to achieve the QoS requirements?Given the source s and the destination d of a unicast dataflow, let l = |L s − L d | denote the source-to-destination distance.Analytical results are obtained under the asymptotic condition of long-range unicasting, l → ∞, where the justification can be that long-range unicasting presents the worst-case scenario of the network performance under practical interests.On the other hand, concise analytical formulation of the QoS metrics can be obtained under this long-range asymptotic condition, which can provide analytical insights to previous questions.

Analytical models
In the performance analysis, models about node distribution, wireless channel, power consumption, bandwidth availability, and the IDLE probability are adopted.Return the request status of moduleRequest(), that is, Permitted or Denied; From system to wireless link GetState(&State) Get the current state.
From system to wireless link

Node distribution model
The node distribution is modeled as a two-dimensional (2D) Poisson process [29], with the node density λ.That is, given an area A of the size |A| in the field, the number of nodes in the area follows Poisson distribution with the parameter λ|A|.The Poisson modeling can be typical for random node distribution and/or random mobility.

Wireless channel model
Given an arbitrary transmitting node n, and a receiving node m, the successful transmission probability, in general, is a function of the transmitting power P t , the radio data rate R, and the distance ζ = |L m − L n |.We denote this function as For example, if small-scale Rayleigh fading [30] is assumed, the channel model can be given by In ( 2), G is a propagation-loss constant; N 0 is the receivernoise power spectrum density; α is the path loss component in wireless channel [30]; B is a threshold constant representing the receiver sensitivity; and ξ is a unit-mean exponential random variable.This model will also be utilized to obtain the numerical results in Section 5.

Power consumption model
Referring to the state diagram in Figure 3, the node power consumption at the IDLE state is denoted by P I , that is, the low-power monitoring of the polling tones; the power consumption at the states of "Send DATA", "Send RTS", "Send CTS", and "Send ACK", is modeled by where P S is the transmitter circuits power consumption; β is a constant decided by the RF power amplifier efficiency [31]; and P t is the transmitting power as defined perviously.
The node power consumption in other states or time is modeled by a constant P R , which denotes the receiving (or idle listening) power consumption.

Bandwidth availability and the IDLE probability
When initiating a unicast module, we assume that the node can always find an available channel by the cognitive radio without significant delay.The IDLE probability is the probability that a given node is in the IDLE state (shown in Figure 3), and can be engaged in a unicast wireless link being initiated by other nodes around it.We assume that there is a fixed (or lower bounded) IDLE probability "ρ", in the analysis.With limited network capacity and arbitrary traffic load, these two assumptions need to be ensured by an appropriate call admission control (CAC) mechanism.On the other hand, given analogous experiences of the popular video streaming on peer-to-peer overlay networks, it could also be appropriate to assume no CAC mechanism, while the traffic volume demanding can always fall below the planned (or unplanned) network capacity.In large-scale wireless networks, the capacity could be of relatively cheap resources, given abundant wireless nodes and the use of unlicensed band, while the traffic load is not necessarily related to the number of wireless nodes.These considerations give the optimization formulations under different types of resource consumptions in Section 4.5.

Other parameters and notations
The packet lengths (bit) of RTS, CTS, DATA, and ACK are denoted by L R , L C , L D , and L A , respectively.T S denotes the time delay in the "channel sensing" state in Figure 3, and T B denotes the time delay in the "Backoff T d " state.We also use the notations E(•) and σ 2 (•) to denote the mean and the variance of a random variable.

Iterative performance analysis
"Iterative performance" can be similar to "single-hop performance."However, with the notation of "iterative performance," we also consider the scenario where the next hop relay node is not found in the transferring branch from "Send RTS" to "channel sensing" of Figure 3.The reason resides in the random node availability, for example, no potential relay candidate is in the IDLE states.Due to this difference, the terminology "iteration" instead of "hop" is used in the following analysis, and "iterative delay" indicates the time delay of an iteration, whereas "iterative energy consumption" indicates the network energy consumption in one iteration.Particularly, consider that there are totally I iterations from the source s to the destination d.Let {n i | 1 ≤ i ≤ I} denote the revelent set of nodes, which initiates the unicast wireless link.Obviously, there is n 1 = s, and n I sends the DATA packet directly to the destination d.The iteration number I is therefore a random variable.

Iterative delay
Let τ i denote the iterative delay at an iteration i. (Other possible models are also discussed in Appendix C, which contribute to the same end-to-end performance analysis results.)If the DATA transmission is completed in the iteration, τ i can be calculated by the formula (L R + L C + L D + L A )/R + T S + T B .Otherwise, if the DATA transmission is uncompleted since no relay is found, τ i can be given by L R /R + T S + T B .Obviously, for the last iteration (i = I), the DATA transmission is completed.
When i < I, the probability of uncompleted DATA transmission depends on the relay candidates availability on the forwarding plane.Due to the Poisson node distribution model with the density λ, and the node IDLE probability ρ, the distribution of relay candidates conforms to an inhomogeneous Poisson process with the density function λ•ρ• f (P t , R, ζ) on the forwarding plane, where ζ indicates the distance to the node n i .Therefore, the "uncompleted" probability is decided by the "equal-zero" probability of a Poisson random variable with the mean λ•ρ•Ω, which is given by e −λρΩ .Ω is always positive, and is given by the integration of the channel model f (•, •, •) on the forwarding plane: According to the above analysis, {τ i | 1 ≤ i < I} can be modeled as a set of random variables with identical distributions, where the expectation E(τ i ) is (5)

Iterative energy consumption
Let i denote the iterative energy consumption at an iteration i. (Other possible models are also discussed in Appendix C, which contribute to the same end-to-end performance analysis results.)If the DATA transmission is not completed in the iteration, the energy consumption of the node n i is (T S +T B )•P R +L R /R•[P S +(1+β)P t ], according to the described models.The energy consumption of one relay candidate is given by T S •P I +L R /R•P R .The number of the relay candidates around n i , which receive the RTS, is a Poisson random variable, with the mean value λ•ρ•Ω.Here Ω can be still given by ( 4), but is integrated on the backward plane, that is, dx dy, instead of the forwarding plane.Therefore, the expected iterative energy consumption under uncompleted DATA transmission is the summation: If the DATA transmission is completed in the iteration, the energy consumption of the node n i is ( The energy consumption of one relay candidate, which is not elected as the next node n i+1 , is still T S •P I +L R /R•P R .The number of the relay candidates is a Poisson random variable with the mean value 2λ•ρ•Ω, including both the forward and the backward planes.On the node n i+1 , the additional energy consumption is Since the DATA transmission is always completed in the last iteration, I is directly obtained from the previous discussion.When i < I, similar to the previous analysis of τ i , the probability of uncompleted DATA transmission is decided by e −λρΩ .Therefore, { i | 1 ≤ i < I} can be modeled International Journal of Digital Multimedia Broadcasting as a set of random variables with identical distributions, where the expectation E( i ) is (6)

Number of iterations
Given all the parameters, the total iteration number I is a random variable decided by the source-to-destination distance l.It is shown in Appendix A that the mean value E(I) and the variance σ 2 (I) are respectively.
In the above, the random variable Λ generally indicates the forwarding distance of one single iteration.Λ is defined with the pdf (probability density function) p Λ (x): where Proved in Appendix B, we point out that the expectation E(Λ) is an increasing function of the node density λ, but is upper bounded, as decided by the channel model f (•, •, •).Furthermore, the variance σ 2 (Λ) decreases to zero with higher λ.Given ( 7) and ( 8), both E(I) and σ 2 (I) can decrease with higher node density λ.These relations can also be intuitively understood, which will be exploited for interpreting the end-to-end performance analysis results next.

End-to-end performance analysis
Analytical results are presented for the maximal throughput Φ, the expected end-to-end delay Υ, the end-to-end delay variance Θ, and the expected network energy consumption per DATA packet Ξ, of a dataflow from the source s to the destination d, under the long-range asymptotic condition l → ∞.The discussion shows how these metrics change with the source-to-destination distance l, as well as the network density λ.

Throughput
The maximal throughput Φ is decided by how many DATA packets can be sent out from the source s to the destination d in a period of time, subject to the condition that this time period is much larger than the packet traveling time.According to previous analysis in Section 4.3.1, the probability of the completed DATA transmission in an iteration is 1−e −λρΩ .Therefore, the expected number of iterations initiated by s for one DATA packet is 1/(1−e −λρΩ ), and the associated time Given the DATA packet length L D , the maximal throughput is thus obtained as It is shown that the maximal throughput Φ can be independent of the source-to-destination distance l.It also increases monotonically with the node density λ, approaching the limit decided by the radio data rate R and the overhead ratio.

Expected end-to-end delay
The end-to-end delay is given by I i=1 τ i .The expected endto-end delay Υ is where (a) is given by the definition; (b) is due to the fact that {τ i | 1 ≤ i < I} are of identical probability distribution [32]; and (c) is obtained directly from ( 5) and (7).Therefore, the expected end-to-end delay Υ increases linearly with the source-to-destination distance l.It also decreases with the node density λ, approaching a bounded limit.

Delay variance
Let I c denote the number of the iterations with completed DATA transmission, that is, the number of wireless hops, and let I u denote the number of the iterations with uncompleted DATA transmission.I c and I u are independent random variables, and I = I c + I u .The delay variance Θ is calculated by the following: where (a) is given by the definition; (b) is due to the analysis in Section 4.3.1, that is, the iterative delay under completed and uncompleted transmissions respectively; (c) is due to the fact that σ 2 (I) = σ 2 (I c ) + σ 2 (I u ); (d) is obtained from the result of σ 2 (I) in (8), as well as the σ 2 (I c ) derived in (A.4) of Appendix A. Therefore, the end-to-end delay variance Θ increases linearly with the source-to-destination distance l.Θ also diminishes to zero with the network density λ, since the component σ 2 (Λ) also decreases to zero with higher λ.

Network energy consumption
The expected network energy consumption for one DATA packet Ξ is where (a) and (b) are given by the definition and the fact that (c) is obtained directly from (7), and the component 1 can be replaced by the result in (6).Therefore, Ξ also increases linearly with the source-to-destination distance l.

Optimizations under QoS constraints
Consider that the dataflow has certain QoS requirements on throughput, expected end-to-end delay, and delay variance, for example, Φ ≥ Φ th , Υ ≤ Υ th , and Θ ≤ Θ th .Here, Φ th , Υ th , and Θ th are the threshold constants representing those QoS constraints, respectively.The optimizations aim to minimize the resource consumption, subject to these QoS constraints.Two types of resource consumptions, including network energy consumption and network capacity, are considered here, which lead to different optimization formulations.As pointed out in Section 4.2.4,these two formulations can be representing different networking scenarios.In capacity limited networks, every node can have cabled power supply.The number of nodes is therefore limited by engineering costs, and the possible use of long-range radio also limits the available bandwidth.On the other hand, in energy limited networks, the nodes can be powered by energy scavenging, for example, solar cells.Therefore, the number of wireless nodes can be large, which can also use short-range radio with vast unlicensed bandwidth.These contribute to large network capacity.Other works with similar considerations have been discussed in [33].

Capacity limited networks
The objective of the optimization is to minimize the transmitting power P t , so as to limit the occupied geographic area of an unicast link.In accordance with the results in [28], this approach can save the use of wireless network capacity.Therefore, the optimization formulation is min P t subject to: Φ ≥ Φ th ; Υ ≤ Υ th ; Θ ≤ Θ th . (15)

Energy limited networks
The objective of the optimization is to minimize the expected network energy consumption per DATA packet Ξ:

Solutions and traffic priority
In order to solve the optimizations in ( 15) and ( 16), the transmitting power P t can be an ideal control knob (optimization variable), for the resource allocation.Both ( 15) and ( 16) can be found as convex optimization problems over P t , where efficient solutions are guaranteed [34].Specifically, as shown in (11), Φ monotonically increases with P t , since Ω (defined in (4)) monotonically increases with P t , provided by any realistic channel model f (•, •, •).As shown in (12), Υ monotonically decreases with P t , since E(Λ) monotonically increases with P t (proved in Appendix B).
International Journal of Digital Multimedia Broadcasting Θ also monotonically decreases with P t in (13), since σ 2 (Λ) monotonically decreases with P t (proved in Appendix B).Moreover, as shown in (14), Ξ can be verified as a convex function of P t , where both E( 1 ) and E(Λ) monotonically increase with P t .Due to the described monotonicity, the convexity of the optimizations in ( 15) and ( 16) can then be obtained, over the control knob P t .
Therefore, we suggest that the transmitting power P t can provide an effective control knob for the tradeoff between the resource consumptions and the real-time QoS requirements.In fact, different levels of P t can be configured for different classes of traffics, for example, voice, video (streaming), or data, which then decide the unicasting "priority".
Further investigation may also include the radio data rate R, so as to generate the problem of joint power and rate control.Unfortunately, the convexity over R cannot be obtained for a general channel propagation model f (•, •, •).Future studies may address the problem on some specific models.

Summary
In summary, the analytical results indicate that all the investigated real-time QoS metrics can improve with larger network scale.In particular, the maximal throughput Φ increases with the network density λ to a predetermined bound, while the expected end-to-end delay Υ decreases with λ, approaching a predetermined bound, as decided by other related parameters.On the other hand, the delay variance Θ falls to zero with higher network density λ, which indicates that the network performance can be made arbitrarily stable, simply by dropping more nodes in the network.For supporting long-range communications over a relatively large number of wireless hops, the analytical results show that the maximal throughput Φ can be independent of the source-to-destination distance, while both the expected delay Υ and the delay variance Θ increase linearly with the sourceto-destination distance.Furthermore, it is identified that the radio transmitting power P t can be an ideal control knob for the tradeoff between the real-time QoS requirements and the resource consumptions, for example, energy consumption or network capacity.
We consider that all these propositions could be intuitively conceivable for large-scale wireless networks.Since the analytical results are obtained under the long-range asymptotic condition l → ∞, they may present the worstcase scenario analysis for practical considerations.In the next section, the simulation results are also provided, showing how these metrics converge.

SIMULATION RESULTS
Some fixed parameters in the simulations are listed in Table 2.In particular, the radio transmission parameters are conforming to the developed test-bed, while others, for example, power consumption and channel model parameters, are typical for indoor short-range radios.For example, under these parameters, the radio range is about 10-15 meters, when the transmitting power P t is 1 mW.Given the source and the destination nodes, separated by the distance l, random Poisson points are generated, for every iteration.The generation of the Poisson points conforms to the density λ•ρ on the 2D plane, which represent the available wireless nodes.1000 DATA packets are then "sent out" consecutively from the source to the destination, according to the described real-time unicast module.The end-to-end delay instance of every DATA packet is then recorded, from which the simulated end-to-end delay expectation and the simulated delay variance are estimated by standard statistical methods.The simulated throughput is also obtained, by dividing the data bulk size 1000•L D with the time consumption of sending out the 1000 DATA packets at the source.In the following figures, we also compare the simulation results to the analytical results side-by-side, where the lower-order terms, for example, o( √ l) or o(l), are neglected in plotting the analytical results.
Figures 4, 5, and 6 show how the QoS metrics change with the source-to-destination distance l, where the node density λ is fixed at 0.1/m 2 .Then Figures 7, 8, and 9 show how the QoS metrics change with the network density λ, where the source-to-destination l is fixed at 160 m.The number of wireless hops in the simulations can be up to 20.All the obtained simulation results match well with the analytical results, as well as the main propositions given in Section 4.6.Especially, Figures 4, 5, and 6 show that closematching also exists in short-range unicasting, although the analytical results were obtained under long-range asymptotic approximation.
Moreover, Figure 10 shows how the expected network energy consumption per DATA packet changes with the transmitting power P t , which is observed as a convex function.It is also observed in Figure 10 that the expected network energy consumption increases linearly with l, and it decreases with λ in the simulation.These simulation results conform to previous performance analysis.

EXPERIMENT RESULTS
In the test-bed (Figure 11), the source and the destination nodes are connected to a laptop computer for the purpose of collecting experiment records.The wireless nodes are based on the cognitive radio prototype (Figure 1), with a stack of two radio boards.The cognitive radio can opportunistically access 3 data channels in 2.4 GHz band, as described in Section 3.2, where each channel has an air-interface data rate 250 kbps as defined in IEEE 802.15.4 [26].The real-time unicast module is implemented in the cognitive radio prototype, with preconfigured address.The packet length parameters, that is, L R , L C , L D , L A , are about the same as those listed in Table 2.The node transmitting power is however fixed at −15 dBm.The experiment results on throughput, end-to-end delay expectation, and delay variance, are obtained by the following methods.(1) Throughput is calculated at the source, where the laptop computer forwards 1000 DATA packets to the source node.Since the source node accepts a new DATA packet from the computer only after the previous one has been sent out, the computer can count the total time expenditure in sending the 1000 DATA packets.The measured throughput is obtained by dividing the bulk size with the time expenditure.(2) On receiving a DATA packet, the destination also forwards the DATA packet to the laptop computer, where the end-toend delay instance of every DATA packet is calculated at the computer by the difference between the packet departure and arrival time.The measured end-to-end delay expectation and  The experiments were conducted in a standard office environment, where external interferences in the utilized 2.4 GHz band can be intensive, for example, from adjacent WLAN hot spots.Two sets of experiments are illustrated in Figure 12.In Set-I, the nodes are placed in a straight line, such that the four configurations correspond approximately to one-hop to four-hop communications.For example, in the second configuration of Set-I, the node placement is such that the chance of direct transmission from s to d is small.In Set-II, more relay nodes are added to every hop being illustrated in Figure 12 Set-I experiments can be comparable to traditional wireless networking where a routing path is predetermined with best available spectrum in every hop.However, in Set-II experiments, data is forwarded from s to d opportunistically without any predetermined path.Figures 13, 14, and 15 show the measurement results of the two-set experiments on throughput, expected endto-end delay, and delay variance, respectively.Evidently, the results of Set-II are strictly better than those of Set-I, beyond the first configuration.In Figure 13, the throughput of Set-I is reduced by more than 50% from the first to the second configuration, that is, from one-hop to twohop, due to the impact of half-duplex radios.A comparable reduction of throughput is not observed in Set-II.In general, the throughput of Set-II changes smoothly with different configurations (number of wireless hops), where the limited reduction can be due to the limited number of nodes.In Figures 14 and 15, it is further observed that the expected end-to-end delay and the delay variance increase linearly with the number of hops (configuration index) in Set-II experiments; whereas they tend to increase much faster in Set-I experiments.

CONCLUSIONS
In the paper, we have implemented a real-time unicast module in the context of large-scale cognitive wireless networking.By opportunistically using both spectrum bandwidth and wireless nodes/radio availability, the most significant result is that the network performance can improve with larger network scale.Specifically, it is shown that the performance dynamics, that is, indicated by dataflow end-to-end delay variance, can fall to zero with higher network density.Furthermore, additional results show that dataflow throughput can be independent of the number of wireless hops or the source-to-destination distance, whereas end-to-end delay expectation and delay variance increase 3), it is shown that E(Λ) increases with λ, approaching a limit decided by the characteristics of g(u), and σ 2 (Λ) decreases with λ, approaching zero.Since g(u), as defined in (10), is a monotonically increasing function of P t , it is also shown that E(Λ) increases monotonically with P t , and σ 2 (Λ) decreases monotonically with P t .

C. MODELS OF THE ITERATIVE DELAY AND ENERGY CONSUMPTION
We point out that more accurate models of the iterative delay τ i (1 ≤ i < I) and the iterative energy consumption i (1 ≤ i < I) can include the distance |L ni − L d |.Under such models, τ i and i will converge to the results in Sections 4.3.1 and 4.3.2,respectively, when |L ni − L d | → ∞.With similar discussion in Appendix A, this will contribute to the same end-to-end analytical results, under the asymptotic condition l → ∞.

Figure 3 :
Figure 3: Wireless link layer state diagram.

Figure 10 :Figure 11 :
Figure 10: Network energy consumption versus transmitting power P t .

Table 1 :
Exemplary primitive functions related to the unicast module.