A low-cost hybrid coordinated checkpointing protocol for mobile distributed systems

Mobile distributed systems raise new issues such as mobility, low bandwidth of wireless channels, disconnections, limited battery power and lack of reliable stable storage on mobile nodes. In minimum-process coordinated checkpointing, some processes may not checkpoint for several checkpoint initiations. In the case of a recovery after a fault, such processes may rollback to far earlier checkpointed state and thus may cause greater loss of computation. In all-process coordinated checkpointing, the recovery line is advanced for all processes but the checkpointing overhead may be exceedingly high. To optimize both matrices, the checkpointing overhead and the loss of computation on recovery, we propose a hybrid checkpointing algorithm, wherein an all-process coordinated checkpoint is taken after the execution of minimum-process coordinated checkpointing algorithm for a fixed number of times. Thus, the Mobile nodes with low activity or in doze mode operation may not be disturbed in the case of minimum-process checkpointing and the recovery line is advanced for each process after an all-process checkpoint. Additionally, we try to minimize the information piggybacked onto each computation message. For minimum-process checkpointing, we design a blocking algorithm, where no useless checkpoints are taken and an effort has been made to optimize the blocking of processes. We propose to delay selective messages at the receiver end. By doing so, processes are allowed to perform their normal computation, send messages and partially receive them during their blocking period. The proposed minimum-process blocking algorithm forces zero useless checkpoints at the cost of very small blocking.


Introduction
In the mobile distributed system, some of the processes are running on mobile hosts (MHs).An MH communicates with other nodes of the system via a special node called mobile support station (MSS) [1].A cell is a geographical area around an MSS in which it can support an MH.An MH can change its geographical position freely from one cell to another or even to an area covered by no cell.An MSS can have both wired and wireless links and acts as an interface between the static network and a part of the mobile network.Static network connects all MSSs.A static node that has no support to MH can be considered as an MSS with no MH.
Wang and Iqbal [30] describe the applications of mobile technology in healthcare.In paper [24], alternative data storage solution for mobile messaging services is provided.Location Management techniques for mobile systems are given in [11,27].Christoph Endres [10] provides a survey of software infrastructures and frameworks for ubiquitous computing.Jayaputera and Taniar [14] propose an approach of mobile query processing when the users location moves from one Base Station to another and the queries cross multi-cells.Cooperative caching, which allows sharing and coordination of cached data among clients, is a potential technique to improve the data access performance and availability in mobile ad hoc networks.N. Chand et al. [7] propose a utility based cache replacement policy to improve the data availability and reduce the local cache miss ratio.
A checkpoint is a local state of a process saved on stable storage.In a distributed system, since the processes in the system do not share memory, a global state of the system is defined as a set of local states, one from each process.The state of channels corresponding to a global state is the set of messages sent but not yet received.A global state is said to be "consistent" if it contains no orphan message; i.e., a message whose receive event is recorded, but its send event is lost.To recover from a failure, the system restarts its execution from a previous consistent global state saved on the stable storage during fault-free execution.This saves all the computation done up to the last checkpointed state and only the computation done thereafter needs to be redone.In distributed systems, checkpointing can be independent, coordinated [3,9,15,29] or quasi-synchronous [2,12].Message Logging is also used for fault tolerance in distributed systems [25].
In coordinated or synchronous checkpointing, processes take checkpoints in such a manner that the resulting global state is consistent.Mostly it follows two-phase commit structure [3,9,15].In the first phase, processes take tentative checkpoints and in the second phase, these are made permanent.The main advantage is that only one permanent checkpoint and at most one tentative checkpoint is required to be stored.In the case of a fault, processes rollback to last checkpointed state.The Chandy-Lamport [6] algorithm is the earliest non-blocking all-process coordinated checkpointing algorithm.In this algorithm, markers are sent along all channels in the network which leads to a message complexity of O(N 2 ), and requires channels to be FIFO.Elnozahy et al. [9] proposed an all-process non-blocking synchronous checkpointing algorithm with a message complexity of O(N).In coordinated checkpointing protocols, we may require piggybacking of integer csn (checkpoint sequence number) on normal messages [5,9,16,19,29].Kumar et al. [18] proposed an all-process non-intrusive checkpointing protocol for distributed systems, where just one bit is piggybacked on normal messages.It results in extra overhead of vector transfers during checkpointing.
The existence of mobile nodes in a distributed system introduces new issues that need proper handling while designing a checkpointing algorithm for such systems.These issues are mobility, disconnection, finite power source, vulnerable to physical damage, lack of stable storage etc.These issues make traditional checkpointing techniques unsuitable to checkpoint mobile distributed systems [1,5,26].To take a checkpoint, an MH has to transfer a large amount of checkpoint data to its local MSS over the wireless network.Since the wireless network has low bandwidth and MHs have low computation power, all-process checkpointing will waste the scarce resources of the mobile system on every checkpoint.Prakash and Singhal [26] gave minimum-process coordinated checkpointing protocol for mobile distributed systems.In minimum-process coordinated checkpointing algorithms, only a subset of interacting process (called minimum set) is required to take checkpoints in an initiation.A process P i is in the minimum set only if checkpoint initiator process is transitively dependent upon it.P j is directly dependent upon P k only if there exists m such that P j receives m from P k in the current checkpointing interval [CI] and P k has not taken its permanent checkpoint after sending m.
A good checkpointing protocol for mobile distributed systems should have low overheads on MHs and wireless channels and should avoid awakening of MHs in doze mode operation.The disconnection of MHs should not lead to infinite wait state.The algorithm should be non-intrusive and should force minimum number of processes to take their local checkpoints [26].In minimum-process coordinated checkpointing algorithms, some blocking of the processes takes place [4,15], or some useless checkpoints are taken [5,16,19].
Acharya and Badrinath [1] gave a checkpointing protocol for mobile systems.In this approach, an MH takes a local checkpoint whenever a message receipt is preceded by the message sent at that MH.This algorithm has no control over checkpointing activity on MHs and depends totally on communication patterns.In worst case, the number of local checkpoints taken will be equal to the number of computation messages, which may lead to high checkpointing overhead.
Cao and Singhal [5] achieved non-intrusiveness in the minimum-process algorithm by introducing the concept of mutable checkpoints.The number of useless checkpoints in [5] may be exceedingly high in some situations [19].Kumar et al. [19] and Kumar et al. [16] reduced the height of the checkpointing tree and the number of useless checkpoints by keeping non-intrusiveness intact, at the extra cost of maintaining and collecting dependency vectors, computing the minimum set and broadcasting the same on the static network along with the checkpoint request.
Koo and Toeg [15], and Cao and Singhal [4] proposed minimum-process blocking algorithms.Neves et al. [22] gave a loosely synchronized coordinated protocol that removes the overhead of synchronization.Higaki and Takizawa [13] proposed a hybrid checkpointing protocol where the mobile stations take checkpoints asynchronously and fixed ones synchronously using the algorithm [15].Kumar and Kumar [20] proposed a minimum-process coordinated checkpointing algorithm where the number of useless checkpoints and blocking are reduced by using a probabilistic approach.A process takes its mutable checkpoint only if the probability that it will get the checkpoint request in the current initiation is high.
Transferring the checkpoint of an MH to its local MSS may have a large overhead in terms of battery consumption and channel utilization.To reduce such an overhead, an incremental checkpointing technique could be used [28].Only the information, which changed since last checkpoint, is transferred to MSS.
In the present study, we design a hybrid coordinated checkpointing algorithm for mobile distributed systems, where an all-process checkpoint is taken after executing minimum-process algorithm for a fixed number of times.By proposing a hybrid scheme, we try to balance the checkpointing overhead and the loss of computation on recovery.We also reduce the piggybacked information onto each computation message.For minimum-process checkpointing, we propose a blocking algorithm, where processes are allowed to perform their normal computation, send messages and partially receive them during the blocking period.
The rest of the paper is organized as follows.We formulate the hybrid checkpointing algorithm in Section 2. The correctness proof is provided in Section 3. In Section 4, we evaluate the proposed scheme.Section 5 presents conclusions.

System model
Our system model is similar to [5,19].There are n spatially separated sequential processes P 0 , P 1 , . . . ,P n−1 , running on MHs or MSSs, constituting a mobile distributed computing system.Each MH/MSS has one process running on it.The processes do not share memory or clock.Message passing is the only way for processes to communicate with each other.Each process progresses at its own speed and messages are exchanged through reliable channels, whose transmission delays are finite but arbitrary.A process in the cell of MSS means the process is either running on the MSS or on an MH supported by it.It also includes the processes of MHs, which have been disconnected from the MSS but their checkpoint related information is still with this MSS.We also assume that the processes are non-deterministic.The i th CI (checkpointing interval) of a process denotes all the computation performed between its i th and (i + 1) th checkpoint, including the i th checkpoint but not the (i + 1) th checkpoint.
When an MH sends an application message, it is first sent to its local MSS over the wireless cell.The MSS piggybacks appropriate information with the application message, and then routes it to the destination MSS or MH.When the MSS receives an application message to be forwarded to a local MH, it first updates the data structures that it maintains for the MH, strips all the piggybacked information, and then forwards the message to the MH.Thus, an MH sends and receives application messages that do not contain any additional information; it is only responsible for checkpointing its local state appropriately and transferring it to the local MSS.

Basic idea
In minimum-process checkpointing, some processes, having low communication activity, may not be included in the minimum set for several checkpoint initiations and thus may not advance their recovery line for a long time.In the case of a recovery after a fault, this may lead to their rollback to far earlier checkpointed state and the loss of computation at such processes may be exceedingly high.Furthermore, due to scarce resources of MHs, this loss of computation may be undesirable.In all-process checkpointing, recovery line is advanced for each process after every global checkpoint but the checkpointing overhead may be exceedingly high, especially in mobile environments due to frequent checkpoints.MHs utilize the stable storage at the MSSs to store checkpoints of the MHs [1].Thus, to balance the checkpointing overhead and the loss of computation on recovery, we design a hybrid checkpointing algorithm for mobile distributed systems, where an all-process checkpoint is taken after certain number of minimum-process checkpoints.The number of times, the minimum-process checkpointing algorithm is executed, depends on the particular application and environment and can be fine-tuned.
In coordinated checkpointing, an ever-increasing integer csn is generally piggybacked onto normal messages [9,29].We propose a strategy to optimize the size of the csn.In order to address different checkpointing intervals, we have replaced integer csn with k-bit CI.Integer csn is monotonically increasing, each time a process takes its checkpoint, it increments its csn by 1. k-bit CI is used to serve the purpose of integer csn.The value of k can be fine-tuned.If we use p-bit CI, we will be able to distinguish only 2 p different CIs and it will be implicitly assumed that no message is delivered after 2 p − 1 CIs.The lower limit of k is '1' which will lead to CI of '1' bit [18].
In the present study, we assume that all-process coordinated checkpoint is taken after the execution of minimum-process algorithm for seven times which requires only three-bit CI.In this case, any delay of a message that extends to more than seven CIs may cause a false checkpoint [18], i.e., it may trigger a checkpoint even if an initiator does not trigger checkpointing activity.Thus, in this algorithm, such delay needs to be avoided.The limit of maximum delay period of a message can be extended to fifteen CIs by using four-bit CI, but it will increase the information piggybacked onto each computation message by 1-bit.By using four-bit CI, we have the option of executing minimum-process algorithm for 3, 7 or 15 number of times before taking an all-process checkpoint.If we use two-bit CI, the maximum delay of a massage should not exceed three CIs, which seems to be unreasonably small in mobile systems.In this case, minimum-process algorithm needs to be executed for three times before taking an all-process checkpoint.
The minimum-process checkpointing algorithm is based on keeping track of direct dependencies of processes.Similar to [4], initiator process collects the direct dependency vectors of all processes, computes minimum set, and sends the checkpoint request along with the minimum set to all processes.In this way, blocking time has been significantly reduced as compared to [15].
During the period, when a process sends its dependency set to the initiator and receives the minimum set, may receive some messages, which may alter its dependency set, and may add new members to the already computed minimum set.In order to keep the computed minimum set intact and to avoid useless checkpoints as in [16,19], we propose to block the processes for this period.We have classified the messages, received during the blocking period, into two types: (i) messages that alter the dependency set of the receiver process (ii) messages that do not alter the dependency set of the receiver process.The former messages need to be delayed at the receiver side.The messages of the later type can be processed normally.All processes can perform their normal computations and send messages during their blocking period.When a process buffers a message of former type, it does not process any message till it receives the minimum set so as to keep the proper sequence of messages received.When a process gets the minimum set, it takes the checkpoint, if it is in the minimum set.After this, it receives the buffered messages, if any.By doing so, blocking of processes is reduced as compared to [4].

Data Structures
Here, we describe the data structures used in the proposed checkpointing protocol.A process on MH that initiates checkpointing, is called initiator process and its local MSS is called initiator MSS.If the initiator process is on an MSS, then the MSS is the initiator MSS.All data structures are initialized on completion of a checkpointing process if not mentioned explicitly.
(i) Each process P i maintains the following data structures, which are preferably stored on local MSS: cci i : Three-bit current checkpointing interval.nci i : Three-bit next checkpointing interval.Maintenance of cci and nci is given below in point (iv).It is the next checkpointing interval, i.e., if P i takes a new checkpoint, the new checkpointing interval will be nci i .tentative i : A flag that indicates that P i has taken its tentative checkpoint for the current initiation.ddv i [ ] : A bit vector of size n.ddv i [j] is set to '1' if P i receives a message from P j such that P i becomes directly dependent upon P j for the current CI.Initially, the bit vector is initialized to zeroes for all processes except for itself, which is initialized to '1'.For MH i it is kept at local MSS.On global commit, ddv[ ] of all processes are updated.In all-process checkpointing, each process initializes its ddv[ ] on tentative checkpoint.Maintenance of ddv[ ] is given in point (vi) below.blocking i : A flag that indicates that the process is in blocking period.Set to '1' when P i receives the ddv[ ] request; set to '0' on the receipt of the minimum set.buffer i : A flag.Set to '1' when P i buffers first message in its blocking period.c state i : A flag.Set to '1' on the receipt of the minimum set.Set to '0' on receiving commit or abort.
(ii) Initiator MSS maintains the following Data structures: minset[ ]: A bit vector of size n.Computed by taking transitive closure of ddv[ ] of all processes with the ddv[ ] of the initiator process [4].Minimum set ={P k such that minset[k]=1}.

R[ ] :
A bit vector of length n.R[I] is set to '1' if P i has taken a tentative checkpoint.

Timer1:
A flag; set to '1' when maximum allowable time for collecting minimum-process global checkpoint expires.

Timer2:
A flag; set to '1' when maximum allowable time for collecting all-process checkpoint expires.

(iii) Each MSS (including initiator MSS) maintains the following data structures: D[ ] :
A bit vector of length n.D[i]=1 implies P i is running in the cell of MSS.

EE[ ]:
A bit vector of length n.EE[i] is set to '1' if P i has taken a tentative checkpoint and A flag at MSS. Initialized to '0'.Set to '1' when some relevant process in its cell fails to take its tentative checkpoint.P in : Initiator process identification.cci in : P in 's cci after it took its tentative checkpoint; matd n * 8 [ ] : A bit dependency matrix to determine whether a message of a particular CI will affect the ddv[ ] of receiver or not; n rows denote the n processes and eight columns denote eight CIs.g chkpt: A flag which is set to '1' on the receipt of (i) checkpoint request in all-process checkpointing or (ii)ddv [ ] request in minimum-process algorithm.chkpt A flag which is set to 1 when the MSS receives the checkpoint request in the minimumprocess algorithm.mss id An integer.It is unique to each MSS and cannot be null.
(iv) Maintenance of Different CIs Initially, for a process, cci and nci are [000] and [001] respectively.When a process updates its CIs, it sets: (i) cci=nci (ii) nci=modulo 8(++nci); for simplicity, we only mention: cci=nci.When a process takes its tentative checkpoint, it updates its CIs.This updating is undone if the checkpointing process is aborted.During minimum-process checkpointing, all such processes, that are not part of the minimum-set, also update their CIs on commit.In this way, when no checkpointing process is going on, all the processes are having the same values of cci.

(v) Maintenance of matd[ ]
Initially, an all-process global checkpoint commit, with cci in = [000], is assumed.On global checkpoint commit with cci in =cci c , matd[ ] is maintained as follows: In this section, we describe, how the ddv vector of a process P i is updated on the receipt of a message or during minimum-process checkpointing.When P i sets its c state, it maintains two temporary bit dependency vectors, ddv1[ ] and ddv2[ ], of length n.These are initialized to all zeroes.The dependencies created during checkpointing process are temporarily maintained in these vectors.On checkpoint completion, these vectors update dependencies of the process.
Suppose, P i receives m from P j , where m.cci is the cci at P j at the time of sending m. minset[ ] is the exact minimum set received along with the checkpoint request.The dependency vectors at P i [ddv, ddv1 and ddv2] are maintained as follows: //P j has taken its checkpoint for the current initiation before sending m else if (matd[j, m.cci]==1) ddv2[j]=1; // Neither P j has taken its checkpoint for the current //initiation nor P j has taken any permanent checkpoint after sending m else receive(m); On Commit or Abort, ddv vector of P i is updated as follows: Case 1.The checkpointing process is aborted.for (k= 0; k<n; k++) The checkpointing process is committed and P i is in the minimum set.
for (k=0; k<n; k++) Case 3. The checkpointing process is committed and P i is not in the minimum set.for (k= 0; k<n; k++)

The proposed minimum-process checkpointing algorithm (a) Checkpoint initiation
The initiator MSS sends a request to all MSSs (MSSs of the mobile system under consideration) to send the ddv vectors of the processes in their cells.All ddv vectors are at MSSs and thus no initial checkpointing messages or responses travels wireless channels.On receiving the ddv[ ] request, an MSS records the identity of the initiator process (say mss id= mss id in ) and initiator MSS, sends back the ddv[ ] of the processes in its cell, and sets g chkpt.If the initiator MSS receives a request for ddv[ ] from some other MSS (say mss id= mss id in2 ) and mss id in is lower than mss id in2 , the current initiation (having mss id= mss id in ) is discarded and the new one (having mss id= mss id in2 ) is continued.Similarly, if an MSS receives ddv requests from two MSSs, then it discards the request of the initiator MSS with lower mss id.Otherwise, on receiving ddv vectors of all processes, the initiator MSS computes minset[ ], sends checkpoint request to the initiator process and sends checkpoint request along with the minset[ ] to all MSSs.

(b) Reception of a checkpoint request
On receiving the checkpoint request along with the minset[ ], an MSS, say MSS j , takes the following actions.It sends the checkpoint request to P i only if P i belongs to the minset[ ] and P i is running in its cell.On receiving the checkpoint request, P i takes its tentative checkpoint and informs MSS j .On receiving positive response from P i , MSS j updates cci i , nci i , resets blocking i , and sends the buffered messages to P i , if any.Alternatively, If P i is not in the minset[ ] and P i is in the cell of MSS j , MSS j resets blocking i and sends the buffered message to P i , if any.For a disconnected MH, that is a member of minset[ ], the MSS that has its disconnected checkpoint, converts its disconnected checkpoint into tentative one and updates its CIs.

(c) Computation message received during checkpointing
During blocking period, P i processes m, received from P j , if following conditions are met: (i) (!bufer i ) i.e.P i has not buffered any message (ii) (m.cci != nci i ) i.e.P j has not taken its tentative checkpoint before sending m (iii) (ddv i [j]=1) ∨ (matd[j, m.cci]= 0)) i.e.P i is already dependent upon P j in the current CI or P j has taken some permanent checkpoint after sending m.
Otherwise, the local MSS of P i buffers m for the blocking period of P i and sets buffer i .On receiving messages, ddv vectors are updated as described in Section 2.3(vi).

(d) Termination
When an MSS learns that all of its processes in minimum set have taken their tentative checkpoints or at least one of its process has failed to checkpoint, it sends the response message to the initiator MSS.
Finally, initiator MSS sends commit or abort to all processes.On receiving abort, a process discards its tentative checkpoint, if any, and undoes the updating of data structures.On receiving commit, processes, in the minset[ ], convert their tentative checkpoints into permanent ones.On receiving commit or abort, all processes update their ddv vectors and other data structures.

Formal outline of the proposed minimum-process algorithm
(a) Actions Taken when P i sends m to P j : send (P i , m, cci i ); (b) Algorithm Executed at the initiator MSS:

All-process checkpointing
Our all process checkpointing algorithm is similar to Elnozahy et al. [8].Initiator MSS sends request to all processes to checkpoint.On receiving the checkpoint request, a process takes the tentative checkpoint if it has not taken the checkpoint during current initiation.After taking a checkpoint, a process updates its CIs.A process, after taking its tentative checkpoint or knowing its inability to take the checkpoint, informs its local MSS.
When a process sends a computation message, it appends its cci with the message.When a process, say P i , receives a computation messagemfrom some other process, say P j, P i takes the tentative checkpoint before processing the message if m.cci equals nci i ; otherwise, it simply processes the message.
When an MSS learns that its all processes have taken the tentative checkpoints successfully or at least one of its processes has failed to checkpoint, it sends the response to the initiator MSS.Finally, initiator MSS sends commit or abort to all MSSs.
On commit, all processes convert their tentative checkpoints into permanent ones and update their data structures.For MHs, MSSs update the data structures.On abort, all processes discard their tentative checkpoints, if any, and undo the updating of data structures.

Handling node mobility and disconnections
An MH may be disconnected from the network for an arbitrary period of time.The Checkpointing algorithm may generate a request for such MH to take a checkpoint.Delaying a response may significantly increase the completion time of the checkpointing algorithm.We propose the following solution to deal with disconnections that may lead to infinite wait state.
When an MH, say MH i , disconnects from an MSS, say MSS k , MH i takes its own checkpoint, say disconnect ckpt i , and transfers it to MSS k .MSS k stores all the relevant data structures and disconnect ckpt i of MH i on stable storage.During disconnection period, MSS k acts on behalf of MH i as follows.In minimum-process checkpointing, if MH i is in the minset[ ], disconnect ckpt i is considered as MH i 's checkpoint for the current initiation.In all-process checkpointing, if MH i 's disconnect ckpt i is already converted into permanent one, then the committed checkpoint is considered as the checkpoint for the current initiation; otherwise, disconnect ckpt i is considered.On global checkpoint commit, MSS k also updates MH i 's data structures, e.g.,ddv[ ], cci etc.On the receipt of messages for MH i , MSS k does not update MH i 's ddv[ ] but maintains two message queues, say old m q and new m q, to store the messages as described below.
On the receipt of a messagem for MH i at MSS k from any other process: add (m, new m q); // keep the message in new m q else add( m, old m q); On all-process checkpoint commit: Merge new m q to old m q; Free(new m q); When MH i , enters in the cell of MSS j , it is connected to the MSS j if g chkpt j is reset.Otherwise, it waits for g chkpt j to be reset.Before connection, MSS j collects MH i 's ddv[ ], cci, new m q, old m q from MSS k ; and MSS k discards MH i 's support information and disconnect ckpt i .MSS j sends the messages in old m q to MH i without updating the ddv[ ], but messages in new m q, update ddv[ ] of MH i .

Example
We explain our minimum-process checkpointing algorithm with the help of an example.In Fig. 1, at time t 1 , P 1 initiates checkpointing process and sends request to all processes for their ddv vectors.During the blocking time of a process, selective messages are buffered as follows.P 2 processes m 0 , because, P 1 has taken permanent checkpoint after sending m 0 .P 2 processes m 6 , because, ddv 2 [3] is already 1 due to receive of m 3 .P 2 buffers m 7 , because, ddv 2 [4] is 0 due to non-receipt of any message from P 4 during current CI.P 2 buffers m 8 to keep the proper sequence of messages received.ddv 4 [5] equals 1 due to m 4 , therefore, P 4 processes m 9 .Similarly, P 5 processes m 10 , because, ddv 5 [4] equals 1 due to m 5 .P 5 buffers m 13 , because, P 3 has taken a new checkpoint before sending m 13 and P 5 has not received the checkpoint request from P 1 .
At time t 2 , P 1 receives the ddv[ ] from all processes [not shown in the figure], computes minset[ ] [which in case of Fig. 1 is {P 1 , P 2 , P 3 }], sets cci 1 =nci 1 , sends checkpoint request along with the minset[ ] to all processes, and takes its own tentative checkpoint.When P 2 gets the checkpoint request, it finds itself a member of the minset [ ].It takes the following actions: (i) take its own tentative checkpoint, (ii) set cci 2 =nci 2 , (iii) send the response to P 1 [not shown in the figure], (iv) process the buffered messages, i.e., m 7 and m 8 .When P 5 receives the checkpoint request, it is not a member of theminset[ ]; therefore, it does not checkpoint but processes the buffered message, i.e., m 13 .At time t 3 , P 1 receives responses, decides to commit or abort the checkpointing activity, and sends abort or commit request to all processes.

Multiple concurrent initiations
We point out the following problems in allowing concurrent initiations in minimum-process checkpointing protocols, particularly in case of mobile distributed systems: (i) If P i and P j concurrently initiate checkpointing process and P j belongs to the minimum set of P i , then P j 's initiation will be redundant.Some processes, in P j 's minimum set, will unnecessarily take multiple redundant checkpoints.This will waste the scarce resources of the mobile distributed system.(ii) In case of concurrent initiations, multiple triggers need to be piggybacked on normal messages [23].Trigger contains the initiator process identification and its csn.This leads to considerable increase in piggybacked information.
Concurrent initiations may exhaust the limited battery life and congest the wireless channels.Therefore, the concurrent executions of the proposed protocol are not considered.

Handling failures during checkpointing
Since MHs are prone to failure, an MH may fail during checkpointing process.Sudden or abrupt disconnection of an MH is also termed as a fault.Suppose, P i is waiting for a message from P j and P j has failed, then P i times out and detects the failure of P j .If the failed process is not required to checkpoint in the current initiation or the failed process has already taken its tentative checkpoint, the checkpointing process can be completed uninterruptedly.If the failed process is not the initiator, one way to deal with the failure is to discard the whole checkpointing process similar to the approach in [15,26].The failed process will not be able to respond to the initiator's requests and initiator will detect the failure by timeout and will abort the current checkpointing process.If the initiator fails after sending commit or abort message, it has nothing to do for the current initiation.Suppose, the initiator fails before sending commit or abort message.Some process, waiting for the checkpoint/commit request, will timeout and will detect the failure of the initiator.It will send abort request to all processes discarding the current checkpointing process.
The above approach seems to be inefficient, because, the whole checkpointing process is discarded even when only one participating process fails.Kim and Park [17] proposed that a process commits its tentative checkpoints if none of the processes, on which it transitively depends, fails; and the consistent recovery line is advanced for those processes that committed their checkpoints.The initiator and other processes, which transitively depend on the failed process, have to abort their tentative checkpoints.Thus, in case of a node failure during checkpointing, total abort of the checkpointing is avoided.

Correctness proof
The correctness proof for the proposed minimum-process checkpointing algorithm is as under: Let GC i = {C 1,x , C 2,y , . . ., C n,z } be some consistent global state created by our algorithm, where C i,x is the x th checkpoint of P i .Theorem 1.The global state created by the i th iteration of the checkpointing protocol is consistent.
Proof: Let us consider that the system is in consistent state when a process initiates checkpointing.The recorded global state will be inconsistent only if there exists a message m between two processes P i and P j such that P i sends m after taking the checkpoint C i,x , P j receives m before taking the checkpoint C j,y , and both C i,x and C j,y are the members of the new global state.We prove the result by contradiction that no such message exists.We consider all four possibilities as follows: Case I: P i belongs to minimum set and P j does not: is the checkpoint taken by P i during the current initiation and C j,y is the checkpoint taken by P j during some previous initiation i.e.C j,y → C i,x .Therefore rec(m) → C j,y and C i,x → send (m) implies rec(m) → C j,y → C i,x → send (m) implies rec(m) → send (m) which is not possible.'→' is the Lamport's happened before relation [21].
Case II: Both P i and P j are in minimum set: Both C i,x and C j,y are the checkpoints taken during current initiation.There are following possibilities: (a) P i sends m after taking the tentative checkpoint and P j receives m before receiving request for dependency: Any process can take the checkpoint only after initiator receives the dependencies from all processes.Therefore a message sent from a process after taking the checkpoint can not be received by other process before getting the dependency request.
(b) P i sends m after taking the tentative checkpoint and P j receives m after getting the dependency request but before taking the checkpoint: In this case, following condition will be true at the time of receiving m: (blocking j ) && (m.cci=nci j ).Therefore, m will be buffered at P j , and it will be processed only after P j takes the tentative checkpoint.(c) P i sends m after commit and P j receives m before taking tentative checkpoint: As P j is in the minimum set, initiator can issue a commit only after P j takes tentative checkpoint and informs initiator.Therefore the event rec(m) at P j cannot take place before P j takes the checkpoint.
Case III: P i is not in minimum set but P j is in minimum set: Checkpoint C j,y belongs to the current initiation and C i,x is from some previous initiation.The message m can be received by P j : (i) before receiving request for dependency (ii) after receiving request for dependency but before taking the checkpoint C j,y If m is received during above (i), P i will be included in the minimum set.If m is received during (ii) above, P j will process m, before taking the tentative checkpoint, if any of the following conditions is true: a. ddv j [i]=1.In this case P i will also be included in the minimum set.b. (matd[j, m.cci]= 0).This is possible only if P i has taken some permanent checkpoint after sending m.In that case, m is not an orphan message.
Case IV: Both P i and P j are not in minimum set: Neither P i nor P j will take a new checkpoint, therefore, no such m is possible unless and until it already exists.
Theorem 2. Checkpointing Algorithm terminates in finite time.
Proof: When initiator initiates a new checkpoint, the initiator and other processes take the following steps: -Initiator asks all MSSs to send the ddv vectors of processes in their cells.All MSSs send the same.
-Initiator computes the minimum set and sends it to all MSSs along with checkpoint request.
-All nodes that are members of minimum set take tentative checkpoints and inform the initiator.If the process is at MH, then the MH may be: disconnected, changing the cell or connected.In the first case, the disconnected checkpoint of MH is used and the last MSS converts this checkpoint to tentative on behalf of MH.In second case, the checkpoint request is delayed and MH takes the checkpoint in the new cell.In third case, MH takes the checkpoint as it is still connected.The MSS that have disconnected checkpoints or the tentative checkpoints of MHs, inform the initiator.-After getting response from all processes/MSSs, the initiator sends commit message to all the processes.-The processes convert their tentative checkpoints into permanent ones after receiving the commit message from the initiator.
All nodes will complete above steps in finite time unless a node is faulty.If a node in the minimum set becomes faulty during checkpointing, the whole of the checkpointing process is aborted (see Section 2.10).Hence, it can be inferred that the algorithm terminates in finite time.

Evaluation of the protocol
Our protocol is a hybrid of all-process and minimum-process coordinated checkpointing schemes.We have also formulated a minimum-process checkpointing algorithm that can be applied independently by using integer csn in place of k-bit CI.Therefore, we evaluate our minimum-process algorithm and the hybrid algorithm separately.

Evaluation of the proposed minimum-process checkpointing algorithm 4.1.1. Computation of average blocking time and average number of messages blocked
The mobile distributed system considered has N MHs and M MSSs.Each MSS is a fixed host that has wired and wireless interface.The two MSSs are connected using a 2Mbps communication link.Each MH or MSS has one process running on it.The length of each system message is 50 bytes.The average delay on static network for sending system message is (8*50*1000)/(2*1000000) = 0.2 ms.The blocking time is 2*0.2 = 0.4 ms.In the proposed algorithm, selective incoming messages at a process are blocked during its blocking period.We consider the worst case in which all incoming messages are blocked.In Cao-Singhal [4] algorithm, a process can neither send nor receive any messages during its blocking period.The number of messages blocked at a process during its blocking period depends upon the message sending rate and blocking period and are shown in the Table 1.
The average blocking period of a message in both the algorithms is 0.4/2 = 0.2 ms.Hence, the number of messages blocked in our algorithm is less than half the number of messages blocked in the Cao-Singhal [4] algorithm, which has got the minimum blocking time of all the existing minimum-process blocking algorithms.

Performance of the proposed minimum-process algorithm
We use the following notations for performance analysis of the algorithms:

Comparison with other algorithms
The Koo-Toueg [15] algorithm is a minimum-process coordinated checkpointing algorithm for distributed systems.It requires processes to be blocked during checkpointing.Checkpointing includes the time to find the minimum interacting processes and to save the state of processes on stable storage, which may be too long.Therefore, this extensive blocking of processes may significantly reduce the performance of the system in mobile environments where some of the MHs may not be available due to disconnections.Each process uses monotonically increasing labels in its outgoing messages.In Koo-Toueg algorithm [15]: (i) only minimum number of processes take checkpoints (ii) message overhead is N mh *( 6C wl + C search ) (iii) Blocking time is N mh (T ch + T search + 4T wl ) [Refer Table 2].Message overhead and blocking time is on significantly higher side in comparison to our minimum-process algorithm.
In Cao-Singhal algorithm [4], blocking time is reduced significantly as compared to [15].Every process maintains direct dependencies in a bit array of length n for n processes.Initiator process collects the direct dependencies and makes a set of interacting processes (S forced ) which need to checkpoint along with the initiator.After sending its dependencies to the initiator and before receiving S forced , a process remains in the blocking state.During blocking period, processes can do their normal computations but algorithm, the average number of processes that take checkpoints in an initiation is slightly greater than the minimum required; but it reduces the loss of computation on recovery.The average message overhead in the proposed protocol is slightly less than [4,19], but greater than [9] [Refer Table 3].In coordinated checkpointing, an integer csn is generally piggybacked on normal messages [5,9,16,19].In the algorithm [4], no information is piggybacked on normal messages.In the proposed algorithm, k -bit CI is piggybacked on normal messages.In the present study, we have taken k = 3. Concurrent executions of the algorithm are allowed in [5].W. Ni et al. [23] have shown that this algorithm [5] may lead to inconsistencies during concurrent executions.

Conclusion
We have designed a coordinated checkpointing algorithm which is a hybrid of minimum-process and all-process algorithms.The number of processes that take checkpoints is minimized to avoid awakening of MHs in doze mode of operation and thrashing of MHs with checkpointing activity.Further, it saves limited battery life of MHs and low bandwidth of wireless channels.Moreover, to avoid greater loss of computation in case of a recovery after a fault, an all-process checkpoint is taken after executing minimum-process checkpointing for a fixed number of times, which, in fact, can be fine tuned.Checkpointing overhead in the proposed scheme is slightly greater than the minimum-process checkpointing but is far less than the all-process coordinated checkpointing.We have introduced the k-bit sequence numbers instead of ever increasing integer csn that is piggybacked on normal messages.This also leads to reduction in the communication overhead.We have also reduced the blocking of processes during checkpointing.
no checkpointing going on { if ( matd[j, m.cci]==1) ddv[j]=1; //P i becomes dependent upon P j after receiving m else receive(m); // P j has taken some permanent checkpoint after sending m; no ddv[ ] is //updated } else if ( blocking i ==1) receive(m); // P i is in blocking period; no ddv[ ] is updated; //selective messages are buffered during this period [Refer Section 2.4(c) and 2.5(c)] else if ((tentative i 1.If the checkpoint initiator process, say P in , runs on an MH, it sends the checkpoint initiation request to its local MSS, say MSS in .2. if (g chkpt) { discard the checkpoint initiation request; inform initiator; exit}.// some global checkpoint recording is already going on 3. MSS in sends request to all MSSs for ddv vectors; set g chkpt in .4. On the receipt of request to send ddv vectors from some other process, say P k : if (P k .ID> P in .ID) {discard P in 's initiation; honor P k 's request; exit;} i : 1. Upon receiving a tentative checkpoint request: -Take a tentative checkpoint; -Send the response to local MSS; 2. On receiving Commit( ): if (tentative i ) { {discard old permanent checkpoint, if any; convert the tentative checkpoint into permanent one;} 3. On receiving Abort ( ): if (tentative i ) {discard the tentative checkpoint;}

Table 1 A
comparison of average number of messages blocked during checkpointing −7 4*10 −6 4*10 −5 4*10 −4 4*10 −3 Cao-Singhal algorithm [4] 8*10 −7 8*10 −6 8*10 −5 8*10 −4 8*10 −3 During the time, when an MSS sends the ddv[ ] vectors and receives the checkpoint request, all the processes in its cell remain in blocking period.During the blocking, a process can perform its normal computations, send messages and partially receive them.In worst case, blocking period of a process is 2T st .The Synchronization message overhead:In worst case, it includes the following:The initiator MSS broadcasts send ddv[ ], take checkpoint() and commit() messages to all MSSs: 3C bst .The checkpoint request message from initiator process to its local MSS and its response: 2C wireless .All MSSs send ddv[ ] of their processes and response to checkpoint request: 2N mss * C st .MSSs send checkpoint and commit requests to relevant processes and receive response messages: 3N mh * C wl .Total Message Overhead (say MOH minp ): 3C bst + 2C wireless +2N mss *C st + 3N mh * C wl .Number of processes taking checkpoints: In our algorithm, only minimum number of processes is required to checkpoint.