Skip to main content
Advertisement
Browse Subject Areas
?

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here.

  • Loading metrics

Tracking the dynamics and allocating tests for COVID-19 in real-time: An acceleration index with an application to French age groups and départements

  • Christelle Baunez,

    Roles Writing – review & editing

    Affiliation Aix-Marseille University, CNRS, Institut Neurosciences Timone, Marseille, France

  • Mickael Degoulet,

    Roles Writing – review & editing

    Affiliation Aix-Marseille University, CNRS, Institut Neurosciences Timone, Marseille, France

  • Stéphane Luchini,

    Roles Conceptualization, Formal analysis, Methodology, Writing – review & editing

    Affiliation Aix-Marseille University, CNRS, Aix-Marseille School of Economics, Marseille, France

  • Patrick A. Pintus ,

    Roles Conceptualization, Formal analysis, Methodology, Writing – review & editing

    patrick.pintus@univ-amu.fr

    Affiliation Aix-Marseille University, CNRS, Aix-Marseille School of Economics, Marseille, France

  • Miriam Teschl

    Roles Conceptualization, Formal analysis, Methodology, Writing – review & editing

    Affiliation Aix-Marseille University, CNRS, EHESS, Aix-Marseille School of Economics, Marseille, France

Abstract

An acceleration index is proposed as a novel indicator to track the dynamics of COVID-19 in real-time. Using data on cases and tests in France for the period between the first and second lock-downs—May 13 to October 25, 2020—our acceleration index shows that the pandemic resurgence can be dated to begin around July 7. It uncovers that the pandemic acceleration was stronger than national average for the [59–68] and especially the 69 and older age groups since early September, the latter being associated with the strongest acceleration index, as of October 25. In contrast, acceleration among the [19–28] age group was the lowest and is about half that of the [69–78]. In addition, we propose an algorithm to allocate tests among French “départements” (roughly counties), based on both the acceleration index and the feedback effect of testing. Our acceleration-based allocation differs from the actual distribution over French territories, which is population-based. We argue that both our acceleration index and our allocation algorithm are useful tools to guide public health policies as France might possibly enter a third lock-down period with indeterminate duration.

1 Introduction

The current COVID-19 pandemic not only caught most of the world by surprise, it also highlighted the crucial aspect of uncertainty under which all societies live since the first mentioning of the new viral infection at the end of 2019. This uncertainty however not only concerns the novel SARS-Cov-2 pathogen and how it spreads, its effects on human health, the best possible treatments, whether current vaccines will be effective agains variants, to just cite some of the concerns. There are at least two more uncertainties which have received less attention. One is parameter-uncertainty of typical SIR (Susceptible-Infected-Removed)-models that are widely used to predict the evolution and consequences of the pandemic, and the other is data-uncertainty. The Spanish newspaper “El Pais” has investigated the data-problem in Spain for example. See https://english.elpais.com/society/2020-06-24/the-problems-spains-outdated-data-methods-have-caused-during-a-21st-century-pandemic.html?rel=mas. While a continuous effort is put into remedying and improving data collection, because this is, after all, the first and foremost important source of empirical knowledge on which all else is built, parameter-uncertainty is intrinsic to modelling strategies and thus much less prone to any easy solutions (see [1] for an example). A key point of this paper therefore is the premise that given these uncertainties, what is of crucial importance to know, here and now, for public health specialists and decision makers and other people alike, is whether harm is accelerating or whether measures that are put in place to control the pandemic are contributing to curbing the spread and thus to decelerate harm (see [2]). In the case of a pandemic, harm can be defined, for example, as the number of confirmed cases. However, simply plotting the number of positive cases over time, as it has been done on many governmental sites, is certainly not enough to understand the dynamics of harm. For example, the knowledge of positive cases depends on testing and hence cases will very much depend on the underlying testing strategy put in place, or lack thereof. But this is exactly our point: to say something meaningful, we need to put cases and tests in relation to each other. That is, to understand whether harm, defined as the number of cases, is accelerating or decelerating, we need to plot cumulated cases against cumulated tests and not to consider them separately.

Although this might be seen as just another tool to visualize the data, we show that such a way to organize them is extremely useful to uncover an index of acceleration, which turns out to be a simple and plain elasticity at the end date of the sample. It tells us how much additional cases are detected given additional tests, in percentage of the total of both variables attained at the date for which one has the last vintage of data. In other words, the “acceleration index” we advocate is the ratio of the growth rate of cumulated over that of cumulated tests. We argue that updating such an acceleration index in real time provides very valuable information and is an essential tool to apprehend the pandemic uncertainties with which we currently live. It is also dealing with parameter-uncertainties and this simply because it does not use any. Instead of engaging in assumptions and probability attributions to ex-ante uncertain events, we use the data that is currently available, which is the best we have, to shed light onto the dynamics of the pandemic and the harm it produces. Our aim in this paper, to be clear, is not to provide a detailed analysis of the pandemic dynamics but to clarify what can be done with available data, in particular to guide public health policy. Our paper is also of conceptual nature, proposing a new way of apprehending and looking at uncertain events (see [3]). We present a real-time operational tool to complement epidemiological SIR models, which are limited in their insights from an ex-ante perspective.

In what follows, we present in more details our novel indicator to track the pandemic, the acceleration index, and its components (section 2.1). Based on this indicator, we show that after a period of deceleration since the end of the first lock-down in France in mid-May, using the data provided by the French Public Health Agency (Agence Santé Publique France), we can observe acceleration emerging from about July 7 onwards (section 3.1), which is about two weeks earlier than other indicators, for example the positivity rate, which was introduced by the French Public Health Agency over the summer to assess the available data. We focus on the post-lockdown period in France—starting May 13, 2020—since too few tests had been performed before that date. As a consequence, the estimated numbers of positives are not reliable before the end of the general lock-down. This is probably the case for many other countries, but not for a few others like South Korea. In a period where time counts and early indications of acceleration should be used as warnings to rapidly prepare health care facilities and to reconsider public health measures in view of increasing the capacities to trace and thus to limit the transmission of the virus, this is essential information and thus an important advantage of our indicator.

Another advantage of our acceleration index is that, being scale-free, it can be used to zoom in on particular groups or geographical areas (section 3.1). Indeed, the indicator can be more helpful the more fine-grained the level at which it is applied. We present a first analysis of the pandemic dynamics for different age-groups and regions in France. Regarding age groups, what we find is that the acceleration over the last two months has been driven more strongly by the age group of people older than 59 than by the age groups between 10 and 28. In particular, the elderly aged 79 or more have not been preserved from this new acceleration. This is important information as it seems that it is not professional or school and educational activities that drive the pandemic’s acceleration. It raises the important and serious question of why harm accelerates exactly at the age groups which are most at risk. On the other hand, it shows that one of the hot-spots for virus transmission at the beginning of the pandemic, nursing homes, had not quite adapted to the new living constraints.

As for the spatial distribution of the dynamics, we present maps with French “départements” (roughly counties) which indicate the evolution of the differences in acceleration across France (section 3.1) from about the period of the end of the first lock-down in mid-May, to the beginning of the second lock-down, end of October. We also look more closely at the acceleration index for the four largest “départements” (Bouches-du-Rhône with Marseille, the North with Lille, Paris and Rhône with Lyon) and compare in particular the acceleration within these areas of the two age groups which are facing a greater acceleration over the last two months of the sample period than the overall acceleration in France.

Finally, based on our indicator, we propose an algorithm for an endogenous testing strategy that takes into account the differences in acceleration across space and age groups (section 2.2). Any test strategy combines two aspects: both information about the virus circulation and the expected feedback that testing implies, in terms of isolation and contact tracing most importantly. We propose a parsimonious algorithm to allocate tests across age groups and space, based on both our acceleration index and the average positivity rate, and on the extent to which tests reduce the virus propagation. Because the latter feature can never be measured with certainty, we argue that any allocation of tests should also reflect the beliefs of (benevolent) public health authorities and experts about it, and we add a parameter to the analysis to take into account this belief dimension explicitly. We show that our proposed acceleration-based allocation of tests differs from the actual distribution of tests, which has essentially been determined by population size (section 3.2). But our analysis shows that population size is not necessarily the criterion that affects acceleration, at least not all the time. Hence, to the extent that testing is accompanied by contact tracing and efficient isolation, this observation means that allocating tests according to where the virus accelerates the most would be a better way to control the pandemic.

2 Materials and methods

2.1 Daily and average positivity rates, and the acceleration index

Accurate and reliable information about how pathogens spread over space and time is of first-order importance to fight epidemics and to properly design efficient public health policies. What is striking in the case of COVID-19 is that, at least in North American and European countries, the main indicators to track the dynamics of the pandemic have been rather coarse and are still limited. To go back to the early stages of the pandemic outside China and neighboring countries, from March to the summer of 2020, the main data used by public health authorities and available to the general public to track the pandemic has been summarized by the time series of the number of positive cases over time.

In Fig 1 are shown, in panels (a) and (b) (top row), such numbers for France, starting May 13 which is when the lock-down for the entire country ended. All the data used in this paper is publicly available from the web page “Données relatives aux résultats des tests virologiques COVID-19 SI-DEP” https://www.data.gouv.fr/fr/datasets/donnees-relatives-aux-resultats-des-tests-virologiques-covid-19/. Panel (a) shows the daily number of people who have been tested positive to COVID-19 for each day, while panel (b) shows the daily but cumulated number of such people over time. It is worth noting that this type of graphs was all there was to visualize the dynamics of the pandemic, at least until late in Spring. In the case of France, the agency in charge of producing public data related to COVID-19, Santé Publique France, initially communicated in terms of number of cases over time only. Santé Publique France decided at some point to report also the positivity rate, that is, the ratio of positive cases to tests at weekly frequency. However, as we will argue, even this indicator does not convey enough information and possible early warning signals that public health authorities could rely on.

thumbnail
Fig 1. Numbers of confirmed cases and tests, daily and cumulated, for COVID-19 in France, May 13—October 25, 2020.

Data source: Agence Santé Publique France.

https://doi.org/10.1371/journal.pone.0252443.g001

The problem with the information contained in the graphs of the first row in Fig 1 is that they would be a fine summary of the pandemic’s dynamics if the number of tests performed over a unit of time was roughly constant. In that case, the evolution of the number of new cases, given a constant over time number of new tests, would capture well how fast the virus spreads. However, as shown in panels (c) and (d) in Fig 1 (bottom row), this assumption is far from correct in practice, and this is not only true for France because of several reasons that has concerned most countries. An important reason behind the observed non-constancy of tests over time in most countries is undoubtedly the fact that the pandemic caught most of the world by surprise in the first quarter of 2020. As a consequence, the widespread goal to progressively increase the capacity to perform tests on a large scale has led many countries to increase the number of tests since March 2020. In the period after the end of the generalized lock-down in France, when the WHO message to “test, test, test”—see https://www.who.int/dg/speeches/detail/who-director-general-s-opening-remarks-at-the-media-briefing-oncovid-19-16-march-2020 last accessed April 14th 2020. was becoming effectively received in France, one sees from panel (c) in Fig 1 that the number of tests performed each day is far from constant and has in fact been trending up most of the time, not surprisingly given the level of unpreparedness of the French government at the beginning of the pandemic.

It is obvious that the dynamics of the number of cases is strongly related to the dynamics of performed tests. After all, diagnostics are the only, albeit imperfect, way to confirm that people are positive to COVID-19. Guidelines from the ECDC explicitly state that the presence of SARS-Cov2 should be assessed via tests and that only cases confirmed that way should be counted in the official statistics. See https://www.ecdc.europa.eu/sites/default/files/documents/TestingStrategyObjective-Sept-2020.pdf. However, observing each independently (side by side) may give only a blurring view at best, and even a confusing sense, of the extent to which the pandemic accelerates. A manifestation of how confusing that coarse piece of information can be, if not properly organized, is the statements often voiced that one should not be worried to observe more positive cases as long as more tests are performed as time passes. After all, more tests imply more cases, almost by definition and this basic fact should not be taken at face value to indicate that the pandemic is worsening.

We argue that such reasoning is wrong and that the correct understanding, in terms of measuring the acceleration/deceleration of the pandemic, is gained if a scatter-plot of the number of positive cases against the number of the tests is used in real time, instead of the panels in Fig 1. This is the first contribution of this paper, in our view, and it is presented in Fig 2. Before presenting our preferred way to organize the data, a few remarks are in order.

thumbnail
Fig 2. Scatter-plot of cumulated daily cases against cumulated daily tests for COVID-19 in France at different dates—Data source: Agence Santé Publique France.

Data normalized using min-max re-scaling. The dashed black line represents the diagonal. The slope of colored solid lines in each panel is a visualization of our acceleration index. The dashed colored lines represent the chords associated with previous dates.

https://doi.org/10.1371/journal.pone.0252443.g002

First, we use the cumulated numbers of daily cases and of daily tests as a way to keep track of the size and overall prevalence of the pandemic. Second, our proposed scatter-plot has merit only if it is updated in real time, say, at daily frequency, so as to add information about additional tests and cases. This immediately raises the question of how data should be normalized, given that the number of data points obviously increases with time. We choose a simple and rather innocuous re-scaling method known as min-max normalization, which in our case amounts to divide all historical values by the last data point figure. See [4], section 3.5.2. Broadly speaking, min-max normalization preserves the relationships, such as quantiles for example, among the raw data before normalization. As a result, our (all positive) normalized data values are contained within the unit interval (0, 1) so that our scatter-plot lies in the unit square.

In Fig 2 we present our proposed scatter-plot, which is in our view a very useful way to visualize the dynamics over time of the number of cases in relation to the number of tests. As alluded to above, there is a causality arrow going from the latter to the former, so we plot cases against tests, all in cumulated values. For example, panel (a) of Fig 2 graphs all data points from March 13 to June 13, 2020, each point/dot representing a particular date. On the x-axis, all numbers are between zero and one due to min-max normalization and a particular value gives the fraction of tests performed that day, say, t, out of the total cumulated number of tests performed up to end date T, say. For example, 0.8 means 80% of the total amount of tests cumulated from date 0 (May 13, 2020, in our case) to date T (June 13, 2020). Similarly, along the y-axis we report the cumulated number of cases for each day tT, as a fraction of the total number of cases at date T. All such normalized data points fill in a solid black curve which then represents the actual data over time. Starting at the origin and moving along the solid curve in the north-east direction means moving forward through time, since we use values that add up over time.

The dashed black line in panel (a) of Fig 2, on the other hand, is the first diagonal and it turns out to be an interesting benchmark if one is interested in capturing the acceleration/deceleration of the pandemic. Counter-factually, If the dashed diagonal would represent the real data, that would tell us that the virus spreads according to a linear pattern whereby, say, 20% of the total amount of tests accounts for 20% of the total number of positive cases, and so on. And this coincidence would hold at all dates between zero and T. In other words, the pandemic neither accelerates nor decelerates since a given proportion of tests account for the same proportion of detected cases for all the data period, that is, over time.

Panel (a) of Fig 2 shows that this has in reality not been the case for the period May 13-June 13, 2020, or for all the period covered by our analysis for that matter. More precisely, the solid line, which represents actual data, turns out to be entirely lying above the dashed diagonal, which should be interpreted in the following manner. First, the blue line representing the tangent to the black curve at the end point (1, 1) turns out to have a slope smaller than one (this is easy to see since the slope of the dashed diagonal equals one by definition). This means that between the day right before T and day T, the observed fraction of new tests (out of the total number of tests to this day) has been associated to a smaller fraction of new cases (out of the total of cases to this day). We then want to say that at date T the pandemic is decelerating. Now going backward from date T into the past (that is, moving from the farthest north-east corner toward the origin along the solid curve), one sees the “slope” of the tangent to the solid curve going up as time moves backward. In other words, the slope of the tangent to the solid curve has been decreasing from date 0 to date T, which is an equivalent way of saying that the solid curve is concave: for a given fraction of tests, lower and lower fractions of cases have been detected over time. This indeed captures in a visual fashion the fact that the pandemic has been continuously decelerating between May 13 and June 13. This observation indicates that France has somehow benefited from the lock-down period in the sense that the circulation of the virus, in the aggregate, slowed down.

Unfortunately, this state of affairs was not to persist. Panel (b) in Fig 2 shows, using color coding, that a month later, on July 13, the slope of the solid curve at end date (the green line) was about one—the green line almost coincides with the dashed diagonal. In other words, the pattern of the pandemic over time switched from decelerating to roughly linear, between mid-June and mid-July. Quite importantly, comparing panel (a) to panel (b) suggests that the slope at end date had started to increase and has crossed unity between June 13 and July 13. While comparing panels (a) and (b) suggests the end of deceleration, panel (c), dated August 13, clearly shows that acceleration was then taking place: the slope at end date is about two. In addition, the scatter-plot at later dates reveal that acceleration is still underway, to this day (October 25, 2020).

One important conclusion is that Fig 2 delivers an indicator in real time of the pandemic’s dynamics, in the sense that it helps to visualize whether the pandemic accelerates or decelerates at any date. The property that the pandemic accelerates when the slope of the solid curve at end date is larger than one, and that it decelerates when the slope is smaller than one, is in fact no surprise. As shown in A.1 in S1 Appendix, this slope is essentially an elasticity: its value gives the variation of cases detected by a given variation of tests, in proportion to levels of (cumulated) cases and tests attained at end date. More generally, the (local) elasticity of a given function is the ratio between its derivative and its average value, taken at a particular point (see [5]). The empirical counterpart that we use in this paper is conceptually similar. For example, a value about two as in panel (c) of Fig 2 means that a given proportional increase in the amount of tests accounts for a proportional increase in the number of cases that is twice as much. One can therefore think of this elasticity as an acceleration index: acceleration happens when the elasticity is larger than one, otherwise the pandemic decelerates. In the razor’s edge case when the elasticity is exactly one, a linear pattern emerges so that the pandemic neither accelerates nor decelerates.

In each panel (b)—(e) of Fig 2, we also report the points corresponding to the previous dates associated with the other panels. For example, the blue dot in panel (b) corresponds to the situation at June 13. Similarly, the red dot in panel (d) corresponds to August 13. Interestingly, our scatter-plot helps also visualizing the pandemic’s dynamics in a retrospective fashion, to the extent that one can visually track not only the slope of the derivative to the solid line, but also the slope of the cord that links the origin to the point corresponding to the date under scrutiny and we now turn to this subtle point.

More specifically, although Fig 2 and the depiction of our acceleration index are very helpful to visualize in real-time whether the pandemic accelerates at end date, those two visualization tools have their own limitations and must be refined if one is to do a retrospective analysis of the pandemic using the information in the scatter-plot and not only the elasticity at end point. Mathematically speaking, the normalization and updating of the data means that data points are subject to a rotational homothety over time. For instance, while the end point at T has coordinates (1, 1), it is subject at date T + 1 to a mapping which is a composition of a contraction, due to the new values of cases and tests at end point, and of a rotation, due to the new value of the ratio between the latter. This is why the colored points move over time in Fig 2. The way we see it is that the scatter-plot should be updated in real time to visualize where the pandemic goes in terms of acceleration/deceleration. We can in particular infer each day, from the most recent data point, the slope of the tangent to the solid curve, the value of which is what we called the acceleration as explained above. But such an index by definition combines two types of information: variations (between two successive dates in our example) and levels at final date. To use the notation of A.1 in S1 Appendix and if we denote the elasticity/acceleration index at end date εT, the following decomposition holds: (1) where PT and DT denote respectively the cumulated numbers of positives/cases and diagnosis/tests at date T, and similarly for date T − 1. If we define from Eq (1) the daily positivity rate to be the ratio between additional positives and additional tests, that is, what we have called earlier the tangent’s slope, then the above decomposition relates the elasticity, the average positivity rate and the daily positivity rate. More precisely, the elasticity is defined as the ratio of the daily positivity rate to the average positivity rate, much as the elasticity of a function is the ratio of its derivative to its average value. In other words, the elasticity is the ratio of the growth rate of cumulated cases over that of cumulated tests. This fact suggests that looking at the average positivity rate alone is not informative enough, in the sense that this indicator does not say how much additional tests translate into additional cases, even though it is of course a fine measure of the average detection rate. It is a proxy for the prevalence of the virus in the population, a key parameter for herd immunity (see [6, 7]).

That said, there might be situations in which any pair of those indicators summarize all there is to know, the third indicator being somehow redundant. For instance, the case depicted in panel (a) in Fig 2 is simple in the sense that, at all dates, both the tangent’s slope and the average positivity rate go down over time, and that the former looks, on close inspection, smaller than the latter. From this property, one concludes that the elasticity is smaller than one. This is of course another way to express the property that the solid line is concave (hence above the dashed diagonal since the solid line goes through the origin and the upper-right edge of the unit square). In contrast, panel (c) in Fig 2 reveals that there has been a change in curvature, that is, a reversal from deceleration to acceleration, between June 13 and August 13. This can be seen from the fact that the least recent part of the solid curve is above the dashed line, while the most recent part is below it. In other words, the convex part indicates acceleration of the pandemic, while the concave part indicates deceleration at earlier dates. One difficulty that arises from such a change in curvature/motion is that the daily and average positivity rates may vary in different directions, with ambiguous implications about whether the elasticity stays smaller than one or, on the contrary, becomes persistently larger than one.

In panel (c) of Fig 2, one notices an inflexion point, when the solid curve intersects the dashed diagonal in its interior. At the date when this happens, the tangent’s slope/daily positivity rate starts to rise, whereas it was declining over time before that date. The average positivity rate, however, continues falling right after the inflexion point is attained, and it starts to rise only later. Eventually, both the daily and the average positivity rate do coincide, and when this happens the elasticity equals one. After that date, the elasticity stays larger than one. More specifically, imagine that the elasticity is smaller than unity and that the daily positivity starts increasing while the average positivity rate is still declining over time. It then becomes possible that the elasticity starts rising and eventually crosses the unit value and possibly stays persistently larger than one. The latter situation occurs whenever the daily positivity rate rises faster (or declines more slowly) than the average positivity rate.

Note that all this is of course reminiscent of what happens when the graph of a continuous and increasing function changes curvature, from concave to convex. We therefore conclude that while the colored line’s slope is an estimate of the elasticity at end point, looking back in time at past events from the perspective of date T through the lens of our scatter-plot requires to examine how both the daily and the average positivity rates vary over time, as shown in Fig 3.

thumbnail
Fig 3. Main acceleration indicators—Daily positivity rate (daily cases to daily tests ratio, black line) and average positivity rate (daily cumulated cases to daily cumulated tests ratio, purple line) over time (left panel), and acceleration index (ratio of daily positivity rate to average positivity rate, right panel, blue line) over time, May 13—October 25, 2020.

Dashed horizontal line in right panel represents when acceleration index equals 1. Data source: Agence Santé Publique France.

https://doi.org/10.1371/journal.pone.0252443.g003

2.2 An acceleration-based algorithm to guide test strategies

Test strategies have many dimensions, including where to test and to allocate staff, locating groups at risk. Typically, few tests are available at the outset of a pandemic and mostly symptomatic people are tested. Then increasing the amount of tests allows to include asymptomatic people but also possibly non-targeted groups, so that growth in testing might be lower or larger than growth in cases. We show in this section how to design an algorithm to allocate COVID-19 test resources. We focus, for the sake of illustration, on the spatial distribution of tests based on our acceleration index. Although testing is acknowledged as an efficient tool to curb epidemics, little is known on the context of COVID-19 about how to geographically allocate a limited quantity of test. The importance of testing is most clearly spelled out in an official document that has been produced by the South Korean government for the international community. See “Flattening the curve on COVID-19—How Korea responded to a pandemic using ICT”, dated April 15, 2020.

Imagine now that public health authorities would like to decide how tests should be allocated to region i at date T. According to the decomposition stated in Eq (1), a small increment of the per-period fraction of tests ΔDi would be expected to lead to a proportional increase of positives in region i. The logic of our criterion to allocate tests across regions has two steps. Firstly, we attribute a weight that equals to each region i. Note that the weight wi attributed to each region is the product of two terms: (i) the first term is the average rate of cumulated positives as a fraction of cumulated tests, at end date T, that is, ; (ii) the second term is the acceleration index εT, which measures whether the pandemic accelerates (i.e. εT > 1) or decelerates (i.e. εT < 1). This means that regions that are allocated more tests relative to others are those where either the overall positive rate is larger at end date, the pandemic accelerates more (or decelerates less), or both, relative to others. Second, we propose to allocate to region i a share of national test capacity that is given by: (2) where β is a parameter which we assume takes positive real values: therefore, si goes up with wi so that a region with a larger weight is allocated a larger share.

Our premise here is that any test strategy combines two aspects: information about the virus circulation and the expected feedback that testing implies, in terms of isolation and contact tracing most importantly. We therefore propose a parsimonious criterion to allocate tests across age groups and space, based on both our acceleration index and the positivity rate, and depending on the extent to which tests reduces the virus propagation. Because the latter feature cannot be measured with certainty, we argue that any allocation of tests should reflect the beliefs that public health authorities have about it and we therefore add a parameter to the analysis to take this key dimension explicitly into account. Obviously, we are assuming here a benevolent public health authorities, who act on objective grounds and are not trying to manipulate the indicator for political purposes.

Therefore, the allocation scheme that we propose builds upon the following logic. When β = 0, each and every region is allocated an equal fraction 1/N of the total amount of tests available at the national level. The other extreme configuration occurs when β tends to ∞ and it is easy to check that, in that extreme case, the region with the highest wi is allocated all national tests.

One way to interpret our allocation rule for intermediate values of β is as follows. Parameter β measures the extent to which testing is believed to exert a negative feedback effect on the pandemic, for example because detecting positives entails isolating them and contact tracing, thereby reducing the number of contacts that produce additional contaminations. In the limit case when β = 0, there is no such feedback and a natural benchmark arises, which consists in allocating tests equally across regions. This corresponds to a situation where information about the spatial distribution of the pandemic is maximized, since each and every region has the same input in terms of testing. In this case, the objective is to gather information about where the pathogen is currently circulating. On the other hand, when β tends to ∞, all tests are allocated to the region with the highest wi and this amounts to maximizing the number of positives so as to curb the pandemic, since in this case the negative feedback is believed to be the strongest. In reinforcement learning terminology, that would lead to stop exploring others aera and only to exploit the region with highest wi when the algorithm starts much like a greedy algorithm would do—see [8]. This is however not a desirable property in the case of a pandemic and acceleration weights should always lead to some exploration to gather information about the virus circulation in other areas, such as in ϵ-greedy algorithms. Middle cases happen when β is between zero and infinity, reflecting the belief held the public authorities deciding over the allocation of tests across regions about the strength of the feedback effect. Note that in theory, β could be negative. This would the case, for instance, if positives would have even more contacts with susceptible persons, generating even more contaminations, compared to a situation with no test. In this case, the share of tests ai would go down when wi goes up. Although such a situation is hypothetically possible—think about bio-terrorism—we rule it out for the sake of realism.

3 Results and discussion

3.1 Estimating the acceleration index for age groups and départements in France

In Fig 3 we report the magnitudes of the three terms that can be inferred from the decomposition in Eq (1), over time. Remember that our objective is to detect whether the pandemic decelerates, accelerates or switches between those two types of motion. A striking feature in Fig 3 is that the acceleration index—in panel (b)—leads the average positivity rate—in panel (a)—in the sense that the former starts rising about two weeks before the latter starts rising as well. In other words, the evolution of the acceleration over time appears to be an early warning of a future rise in the average positivity rate. Eventually, in view of the reasoning above, one expects of course that the rising acceleration index crosses the unit value when the (rising) average positivity rate equals the (rising) daily positivity rate. The bottom line is that the decomposition that follows from the scatter-plot in Fig 2 may be useful to detect the early reversal from deceleration to acceleration. In the case of France, the elasticity starts rising about July 7, while the average positivity rate starts rising about two weeks later, on July 23, as indicated by the orange bars in Fig 3. Our acceleration index has, to our knowledge, not been studied before, but many health agencies around the world report the positivity rate over time. In particular, Santé Publique France started, during the second quarter of 2020, to report the weekly positivity rate on top of the number of cases. The first time a rise of such an indicator is noted is July 23, in the “Point épidémiologique hebdomadaire du 23 juillet 2020”, whereas our acceleration index starts rising on July 7 as shown above. This two-week gap may suggest that using daily data rather than weekly data is more appropriate.

To sum up, the important piece of information that one can derive from Fig 3 is that a reliable measure of the pandemic’s dynamics is the elasticity, which we can think of as an acceleration index. This elasticity leads the average positivity rate and is more dynamic than the daily positivity rates, which are other key statistics to follow. In practice, therefore, panels (a) and (b) should be used as real-time indicators of the pandemic’s dynamics. From the acceleration index and average positivity rate then follows the daily positivity rate, which is still useful to translate the transmission dynamics back into levels (as opposed to normalized ratios). In [9], we relate the acceleration index to the reproduction number that is a core concept in the epidemiological literature. We show that the former is a more accurate measure of the pandemic’s dynamics than the latter. See also [10] for a related study of curfew and lock-down in France.

From panel (b) in Fig 3, one infers that the period since the end of the first lock-down in France can roughly be split into four sub-periods. From May 13 to July 7, our acceleration index fluctuates around a value slightly lower than one, indicating that the pandemic is in a deceleration regime. Our indicator suggests that July 7 is the start of an ongoing acceleration regime in which the acceleration index increases, crosses the unit value on July 23, and continues to increase until about mid-August, when our acceleration index equals about 2. In that sense, this second sub-period can be seen as the fact that the pandemic worsens more and more over time. Then a new plateau occurs until around October 1st, after which the acceleration index rises again to reach a value of about 3 on October 25. Of course, the challenge that the country faces as the second lock-down period starts, on October 30, is to reverse that pattern to ensure that the acceleration index starts decreasing and eventually reaches values below one, which would indicate deceleration of the pandemic.

The simple case whereby both cases and tests grow exponentially, as detailed in Section A.2 in S1 Appendix, is perhaps useful to interpret the four sub-periods just described. The first plateau associated with deceleration is compatible with a situation where cases have been growing at a slower rate than tests for a while. In that situation, the acceleration index is roughly constant over time, below unity. Then if the growth rate of positives becomes larger than the growth rate of tests, then the acceleration index starts rising and eventually crosses unity, to converge to a new and higher plateau associated with a value larger than one. Finally the most recent rise, starting early October, can be understood as a new jump of the growth rate of positives. Although the above interpretation stresses changes in the growth rate of positives, what matters more generally is how the ratio of growth rates—that of positives divided by that of tests—varies over time.

In addition, the scatter-plot also directly offers a measure of “global acceleration/convexity”: in panels (d), (e) of Fig 2, the solid curve is located entirely below the dashed diagonal, which indicates that the persistent acceleration somehow “dominates” the earlier and short-lived deceleration. We conjecture that such a global measure of convexity relates to δ-convexity as defined by [11].

Based on the methods described in the previous section, we propose to use the acceleration index, calculated from the end point of our scatter-plot, as a parsimonious way to track the pandemic’s dynamics. In this section, we show how useful such an index is to decompose the evolution of acceleration over age groups, first, and space, in a second step.

In Fig 4, we report the evolution over time of our acceleration index for nine age groups, all aggregated over space. In each panel, the blue line is a replica of panel (b) in Fig 3. For comparison against such an average benchmark, the orange line represents the acceleration index corresponding to the age group of each panel. One striking feature is that acceleration is underway for all age groups except people older than 79 years, as of October 25, 2020. In addition, the acceleration for each age group tracks rather closely that of the average for some groups but much less so or not all for others.

thumbnail
Fig 4. Acceleration index for different age groups, May 13—October 25, 2020.

In each panel, the blue line reproduces panel (b) in Fig 3, the dashed horizontal line represents when acceleration index equals 1, while the orange line corresponds to the age group specific to that panel. Data source: Agence Santé Publique France.

https://doi.org/10.1371/journal.pone.0252443.g004

To start with the former groups, the middle row in Fig 4 shows that people aged 29 to 58 experience acceleration in much the same way as the average over all age groups. This is not the case for younger and older groups. Children aged 10 to 18 and, quite spectacularly, young adults aged 19 to 28 seem to face a noticeable but slower acceleration since summer than the general acceleration. The latter group, in particular, is associated with an acceleration index that is maintained at intermediate levels around 2 in the last quarter of the data sample.

The fact that the share of confirmed cases is the second highest (22.9%) for the [20–29] age group (see Table 1 in A.3 in S1 Appendix for incidence rates among age groups) may contribute to the view that youngsters such as college students have contributed a lot to the resurgence of the virus since last summer. This view does not agree with the data in Fig 4. The age group [0–9] has experienced ups and downs since last summer. In particular, the acceleration index has started to rise again quickly in early September across France, after it went down all the way to unity at the end of summer. The question here of course is whether the rapid increase has to do with the beginning of the academic year and the fact that young pupils did not wear face masks at school. However, another reason may be that younger kids have again been in more contact with their grand-parents, who take care of them some of the time while parents are working since the beginning of the school year. Yet in Fig 4, it is striking that the acceleration index for the age group [69–] has risen importantly since the beginning of August, to attain the highest level at the end of the data sample, around 4! A similar, though slightly less strong rise, can be seen for people aged [59–68]. Hence maybe it is not the young kids who infect their grand-parents, but it is the other way around? Studies suggest that the “engines of the SARS-COV-2 spread” are household transmissions, which may support this hypothesis (see [12]).

Equally remarkable is the group of people older than 79, which has experienced a catching-up in acceleration since mid-August, similar to—though later than—the dynamics of all other age groups. This suggests that the continued presence of COVID-19 does not seem to be under control in specialized nursing homes (the “EPHAD”). Given the large number of deaths among the very elderly at the beginning of the pandemic, this fact raises some serious concern.

Our analysis can also be applied to shed light on the local evolution of the pandemic. In Fig 5, we report in a map of France the acceleration index at the level of administrative division which is intermediate between the city and the region, called “département”. We use a background map that is in the public domain and can be downloaded via https://commons.wikimedia.org/wiki/File:D%C3%A9partements_de_France-simple.svg. All maps in this paper are then constructed by incorporating our own data into this background map. Red entries represent places where the pandemic accelerates (the elasticity is larger than one), while green entries represent decelerating départements. Our map does not necessarily coincide with the map very recently introduced by the European Centre for Disease Prevention and Control (ECDC) for Europe. See https://www.ecdc.europa.eu/en/covid-19/situation-updates/weekly-maps-coordinated-restriction-free-movement. In that map, green-orange and red lights depend on the incidence rate and the test positivity rate of a country but their combination of factors is not dynamic and does not have any clear conceptual foundation such as our acceleration-based index has. Their demarcation criteria between the different colours are set at a certain level, but it is not clear whether that level is sound from the perspective of our acceleration index.

thumbnail
Fig 5. Acceleration index for French départements at different dates.

Data source: Agence Santé Publique France.

https://doi.org/10.1371/journal.pone.0252443.g005

The upper left panel in Fig 5 depicts the level of the acceleration index for each département on May 15, that is, on the third day after the end of the lock-down period in France. While at that date, the pandemic is decelerating over almost all the country, the situation starts to gradually but dramatically change within the next two months and to reverse course over the summer break. Between July 15 and August 15 (panels in second row), the country switch to a situation such that acceleration takes over the country. As of October 25, most département face an acceleration index larger than two, with no sign of reversal.

At the time when France enters a second lock-down period, lessons should be drawn about what went wrong in the period following the end of the first lock-down and we argue that it is important to analyse closely the reasons for local and group-specific increased accelerations. Knowing about local and group specific accelerations can be an important guide in investigating further possible transmission channels and networks, which would then allow to intervene specifically at a network and structural level if needed.

To give an example of such a fine-grained analysis, we dig a little deeper into the heterogeneity that is visible in Fig 5, and the accelerations we spotted in the different age groups. We report in Fig 6 our acceleration index for the four largest départements in terms of population, namely Bouches du Rhône, Rhône, Paris and Nord, which have been allocated more tests than smaller départements as the next section will show. Fig 6 features in panel (a) the dynamics for all age groups combined, while panels (b) and (c) depict that of kids aged [0–9] and of people aged [59–79] respectively.

thumbnail
Fig 6. Acceleration index for the four largest French départements, May 13—October 25, 2020.

In each panel, the dashed horizontal line represents when acceleration index equals 1, the blue line depicts acceleration for all other départements while the colored lines correspond to the département specific to that panel (Bouches du Rhône, Rhône, Paris and Nord, from top line to bottom line). Data source: Agence Santé Publique France.

https://doi.org/10.1371/journal.pone.0252443.g006

In all panels, the blue line stands for the evolution of the acceleration index for all the other départements, while the second colored lines in each row represents the dynamics of acceleration for all age groups in panel (a) and the specific age group in panels (b), (c). Looking first at panel (a), one sees that the dynamics in all four départements, resemble that of the others, with perhaps more volatility in the case of Bouches du Rhône and Rhône. Note, however, that the latter have experienced a period when the acceleration index decreased, roughly from mid-August to mid-September. This is not really the case for Paris and Nord.

In panel (b), we see considerable volatility, largely due to the fact that the absolute numbers of tests and positives are rather low for this age group at the level of the département, the acceleration index for the [0–9] age group hover around a plateau below one from mid-May to early July. What is striking, however, is that the acceleration index drops a lot before the end of August to levels close to unity, only to starts rising sharply again in the first half of September. As we asked before, why is there this rapid acceleration in this age group? Does it have to do with the dynamics of acceleration for the age group of grand-parents, depicted in panel (c). The rapid acceleration in this age-group is especially the case in Paris and Nord, where the acceleration index for the elderly rises continuously from the beginning of August, and may be less so for Bouches du Rhône and Rhône. In contrast, deceleration was pronounced both for kids and for grand-parents before the beginning of summer.

Although the above comments in relation to Fig 5 are merely descriptive, and deliberately so, they illustrate how our acceleration index can be used at any levels of granularity that is allowed by the available data and how it may also guide further analysis about transmission channels to complement contact tracing. An interesting avenue for research would obviously be to perform a panel data analysis, but this is beyond the scope of this paper.

3.2 Comparing actual and acceleration-based allocations of tests

We use Fig 7 to visually compare the allocation of tests, as computed from our algorithm based on the acceleration index, described in section 2.2, to the observed allocation and to the population distribution, as of October 25, 2020. See also [13] for an earlier application to Italy.

thumbnail
Fig 7. Allocation of tests and population in French départements, as of October 25, 2020—Actual in upper panels and allocation of tests derived from our acceleration index in lower panels.

Data source: Agence Santé Publique France.

https://doi.org/10.1371/journal.pone.0252443.g007

In panel (a) and (b), we report the observed shares of tests and of total population in each département, respectively. Comparing the two upper panels reveals that the observed allocation of tests at that date reflects the geographical distribution of population to a large degree: in short, more tests have been done in more populated areas.

Panels (c) and (d) in Fig 7 depict the allocation of tests over French départements, for different weights β put on the acceleration index. Comparing the upper and the lower panels makes clear that the spatial distribution of tests observed at October 25, 2020, has little connection to where the pandemic accelerates, which is somewhat paradoxical to the extent that detecting positives should help to curb the virus circulation. More precisely, while in panel (a) one sees that tests have been concentrated, especially in about roughly three sub-regions (Mediterranean South East, North, and Rhône), panels (c), (d) shows that the allocation proposed by our algorithm based on the acceleration index is somewhat less concentrated or concentrated in different areas. This is particularly clear when β = 1. When β is higher, our algorithm allocates more tests where the acceleration index is larger than unity (and less tests where it is smaller than one) so that the allocation of tests becomes more concentrated but panel (d) shows how more tests should go to the south-east quarter of France. In contrast, panel (a) shows that this is not what has actually happened.

The difference between the actual distribution of tests over départements, which is very much population-driven, and our proposed allocation, which acceleration-centered can be established a it more formally by looking at the following measures. In S1 Appendix A.4, we report in Tables 2–4 in S1 Appendix the numbers behind Fig 7 and we use those to compute two indicators that shed light on the differences between the actual and proposed spatial distributions of COVID-19 tests.

We first carry out a comparison between the different distributions using Jensen-Shannon (JS) divergence, which is based on Kullback-Leibler divergence (see [14]) and is normalized between 0 and 1, to measure similarities between the actual distribution of tests, population shares and our acceleration-based allocation of tests. From Tables 2–4 in A.4 in S1 Appendix, we find that JS divergence between the observed distribution of tests and the distribution of population equals 0.016, hence very small. This confirms our suggestion that relative population size has been an important determinant of the distribution of tests across French départements. In contrast, JS divergence between the observed distribution of tests and our proposed distribution based on acceleration is higher: it equals 0.115 when β = 1 and 0.194 when β = 3. Of course, JS divergence between the distribution of population and our proposed distribution features close values: it equals 0.092 when β = 1 and 0.193 when β = 3. Not surprisingly in view of the above discussion in relation to Fig 7, we conclude that observed and proposed distributions differ in a quantitative sense, but also that putting more weight on acceleration—that is, increasing β—further enlarges thus discrepancy.

Some limitations of the analysis carried out in this paper and avenues for further research are worth mentioning at this point. In virtually all papers in which non pharmaceutical interventions to fight the pandemic are evaluated, virus propagation is measured by some empirical estimates of the reproduction number. Because the latter does not control for time-varying amounts of tests while our acceleration index does, it would be informative to relate both indicators, both in theory and in practice. Although this issue is not addressed here, in [9] we report a proper comparative analysis. An additional limitation of the acceleration index is that it compares the daily positive rate to the average positivity rate, which summarizes the entire “past” in the sense that it uses all available date since the outset of the pandemic. In contrast, reproduction numbers are usually estimated over a given moving time window that relates to the period of infectivity. Understanding the relative merit of measuring virus propagation over a certain time window or over all the relevant past is an important task that is left for further research.

Another set of limitations that our approach admittedly has comes from the fact that it relies on actual data only and abstracts away from any theoretical model like those that have been stressed and used by some governments from early on. Yet, we do so deliberately and this for two reasons. First and foremost, COVID-19 has clearly put at the forefront the importance of admitting that the novelty of a new pathogen has implications for modelling. Any estimates obtained through, say, SIR or any other models, are subject to considerable and often unknown parameter uncertainty. This problem is particularly acute in the face of a novel pathogen, when public health authorities must act swiftly to prevent the virus from spreading in an uncontrolled way. Our analysis offers a real-time analysis that builds only on raw data and is, as such, subject to data measurement issues as any other approaches, but not to parameter uncertainty. Second, our real-time analysis is scale-free and can therefore be used at very granular levels, say cities or counties or age groups, where any model becomes rapidly either untractable or not transparent enough. In fact, the results in this paper could be used to go in the direction of advocating more diversity in the tools that should be used in an emergency crisis, that is, a type of “method averaging” (in the same sense that model-averaging is used) that seems to us necessary to take the best policy actions given the available information. That said, given that our acceleration index is fed with data on cases and tests, some of its limitations relate to the fact that it does not incorporate information about over vs under-reporting, and who exactly is tested and why. For instance, diagnostics may be triggered by public health authorities targeting certain groups, but also by the population itself for a variety of reasons including the protection of family members.

Obviously, the allocation of tests and the pandemic’s dynamics are both endogenous and interactions between the two form a complex system. Our analysis is so far limited to tracking such dynamics and proposing an algorithm to allocated tests across space, according to the acceleration index. As such, it has little to say about the extent to which testing impacts negatively the virus circulation. For instance, it is reasonable to think the weights our algorithm put on acceleration should necessarily be the same in all départements. Perhaps isolation and contact tracing is more efficient in some regions that in others. In addition, additional constraints should arguably be introduced, such as the fact that the number of tests allocated to a département should not be larger than its population. While such constraint is unlikely to bind at the département level, it could at more granular levels at which our algorithm could operate as well. For example, some little populated cities could be allocated too much tests in view of their populations. In that case, our algorithm could easily be amended to account for such constraints, by capping test and reallocating the surplus of test according to the next cities where acceleration is the highest. Despite all these limitations, we still argue that our algorithm to allocate tests according to where the pandemic accelerates could be a valuable input to public health policies, not only in France of course but also in all countries facing COVID-19. Of particular importance, at this point, is how to reallocate the national testing resources and effort within département to specific age groups, and of course when the pandemic will start to decelerate and vanish globally, which one hopes is not too far ahead in time. See [15, 16] for other considerations.

4 Conclusion

The purpose of this paper is to show that plotting the number of cases, of tests and even the positivity rate against time is far from the best way to measure the acceleration/deceleration of the virus. We propose instead a simple yet novel way to look at the data in real time. Our premise is that looking solely at the number of cases over time to measure acceleration of the pandemic is not accurate and hence problematic as a foundation for public health policies. This is because the amount of tests is far from constant per unit of time and this for a number of reasons. We thus argue that a much better understanding is gained by plotting the number of cases against the number of tests, that is, a simple and plain scatter-plot.

From such a scatter-plot we derive an acceleration index, which we propose, is a useful indicator to track the dynamics of pandemics like COVID-19 in real-time. Using French data on confirmed cases and tests for the period between the first and second lock-downs—May 13 to October 25, 2020—our acceleration index shows that the pandemic resurgence can be dated to begin around July 7. Rather surprisingly, it helps to underline the fact that the pandemic acceleration has been stronger than national average for the [9–68] and especially the 69 and older age groups since early September, the latter being associated with the strongest acceleration index, as of October 25. In contrast, acceleration among the [19–28] age group is the lowest and is about half that of the [69–78]. We also propose an algorithm to allocate tests among French départements, based on both the acceleration of the pandemic and the feedback effect of testing. Our acceleration-based allocation differs significantly from the actual distribution over French territories, which is population-based.

The index that we have exposed in this paper could also be used in the following weeks to monitor the resurgence of the pandemic, in France and elsewhere obviously, and to detect positive signs of deceleration and to respond rapidly to new accelerations. The point is that our acceleration index might better track any reversal than looking solely at the average or daily positivity rate. When deceleration is under way, our index gives a very clear target for an all-clear: the acceleration rate has to be lower than one independently of where we look through our “acceleration lenses”, that is a spatial-and-group-based analysis for example. It is also in this sense that our acceleration index can help assessing and modulating health policies to respond to those areas or groups where an acceleration takes place. Arguably, this would help avoiding generalized lock-downs, which are extremely costly, or other policies that affect the general society independently of whether they face an acceleration or not.

Acknowledgments

The authors thank Frédéric Aprahamian and Fabrice Carrat for useful comments on this paper, and the Editor and both referees for suggestions that prompted us to improve the draft.

References

  1. 1. Gurdasani D., Ziauddeen H. (2020): On the Fallibility of Simulation Models in Informing Pandemic Responses. The Lancet, 8(6): E776–E777. pmid:32359419
  2. 2. Taleb N. (2012). Antifragile, things that gain from disorder. Random House.
  3. 3. Degoulet M., Willem L., Baunez C., Luchini S., Pintus P. (2021): Sensitivity to Rare and Extreme Events in Rats: the Black-Swan-Avoidance Bias. Forthcoming bioRΧiv and AMSE working paper.
  4. 4. Han J., Kamber M., Pei J.(2011): Data Mining: Concepts and Techniques. Elsevier.
  5. 5. Arrow K.J., Chenery H.B., Minhas B.S., Solow R.M. (1961): Capital-labor substitution and economic efficiency. Review of Economics and Statistics, 43(3):225–250.
  6. 6. Britton T., Ball F., Trapman P. (2020): A Mathematical Model reveals the Influence of Population Heterogeneity on Herd Immunity to SARS-CoV-2. Science 369 (6505): 846–849. Available at https://science.sciencemag.org/content/369/6505/846
  7. 7. Fontanet A., Cauchemez S. (2020): COVID-19 Herd Immunity: where Are We? Nature Reviews Immunology, 20(10): 583–584.
  8. 8. Sutton R.S., Barto A.G. (2018): Reinforcement Learning, An Introduction. The MIT Press, Cambridge, MA, Second Edition.
  9. 9. Baunez C., Degoulet M., Luchini S., Pintus M., Pintus P., Teschl M. (2020): The Acceleration Index as a Test-Controlled Reproduction Number: Application to COVID-19 in France. MedRχiv preprint 12.01.20241570 available at https://www.medrxiv.org/content/10.1101/2020.12.01.20241570v2.
  10. 10. Baunez C., Degoulet M., Luchini S., Pintus P., Teschl M. (2020): An Early Assessment of Curfew and Second COVID-19 Lock-down on Virus Propagationin France. MedRχiv preprint 11.11.20230243 available at https://doi.org/10.1101/2020.11.11.20230243.
  11. 11. Hyers D. H., Ulam S. M. (1952): Approximately Convex Functions. Proc. Amer. Math. Soc. 3:821–828
  12. 12. Lee E.C., Wada N.I., Grabowski M.K., Gurley E.S., Lessler J. (2020): The Engines of SARS-CoV-2 Spread. Science, 370(6515): 406–407.
  13. 13. Baunez C., Degoulet M., Luchini S., Pintus P., Teschl M. (2020): Sub-National Allocation of COVID-19 Tests: An Efficiency Criterion with an Application to Italian Regions. Covid Economics, 12:192–209.
  14. 14. Kullback S., Leibler R.A. (1951). On Information and Sufficiency. Annals of Mathematical Statistics 22(1): 79–86.
  15. 15. Haleem A., Javaid M., Vaishya R., Deshmukh S.G. (2020): Areas of Academic Research with the Impact of COVID-19. The American Journal of Emergency Medicine, 38(7): 1524–1526. Available at https://doi.org/10.1016/j.ajem.2020.04.022.
  16. 16. Haleem A., Javaid M., Vaishya R. (2020): Effects of COVID-19 Pandemic in Daily Life. Current Medicine Research and Practice 10(2): 78–79. Available at https://doi.org/10.1016/j.cmrp.2020.03.011.