exampletestr—An easy start to unit testing R packages

In spite of the utility of unit tests, most R package developers do not write them. exampletestr makes it easier to start writing unit tests by creating shells/skeletons of unit tests based on the examples in the user's package documentation. When completed, these unit tests test whether said examples run correctly. By combining the functionality of exampletestr with that of covr, having ensured that their examples adequately demonstrate the features of their package, the developer can have much of the work of constructing a comprehensive set of unit tests done for them.


Introduction
Unit tests are automated checks which examine whether a package works as intended. There are many reasons to write unit tests: 1. To identify bugs in your current code. If a test fails, something is not working as expected.
2. To make current package functionality robust to future changes. If you have written good tests, you do not need to worry that changes made to your code have broken existing functionality; the tests will tell you whether or not the original functionality is in tact. In the words of Hadley Wickham, "I started using automated tests because I discovered I was spending too much time refixing bugs that I'd already fixed before." 1 3. Test-driven development (TDD). Some people advocate writing tests before writing code: the tests outline what your code should do and you know your code is working correctly when the tests start passing. The use of TDD achieves the quality control goals of points 1 and 2 above as code is written. This contrasts with retrospective unit testing, which is to write unit tests after everything else is done. 4. To assure others that your code works correctly. The fact that a package is unit tested indicates to potential users that the author has taken care to ensure that their code works as intended. 3. Complete the writing of unit tests, checking your code coverage with package_coverage(type = "tests") %>% shine().
Using this workflow, the developer ensures that their example coverage (the portion of package features covered by documentation examples) is adequate, and simultaneously obtains a reduction in the work required to write comprehensive unit tests.

Discussion
Testing whether documentation examples run correctly (which is what exampletestr is useful for) is just one important part of writing good unit tests. Two points to bear in mind when using exampletestr:

Documentation examples typically showcase a best case
application of the code in a package. Unit testing should go beyond this and assess the execution of edge cases, e.g. the performance of functions when an argument is empty (length zero). exampletestr does not help in this regard.
2. exampletestr promotes a "one test per function" model. However, in many packages, it is necessary to go further than just testing whether each function performs well in isolation. For instance, it is good to test function composition: whether one obtains the expected results when applying two or more functions, one after another. The data presented can be found at https://github.com/rorynolan/ exampletestr/tree/master/analysis, along with a file showing how to reproduce that data.

Author contributions
Rory Nolan conceived the idea, created the package and wrote the paper. Sergi Padilla-Parra supervised the coding of the package, helped to test it and helped to write the paper.

Competing interests
No competing interests were disclosed. I read the manuscript and found it was clearly written. I also successfully executed the examples in the package vignette. In general, I believe the authors have prepared a useful package and have described it well.

Grant information
I have however two important points I would like to raise.
My first point is that I think it would be useful for the authors to provide a better rational for unit testing. In my opinion, the current version of the text is obvious to those that already know about and practice unit testing. However, the goal of this work is to appeal to the many that haven't adopted unit testing in their standard development yet. For those, it might be useful to provide more background and to emphasise standard development yet. For those, it might be useful to provide more background and to emphasise the important of unit testing.
My second point is that by focusing on code from examples, some of the most important unit tests are systematically missed out. Indeed, one generally documents a typical, default, application of best case the code/package in the documentation files. One could argue that the goal of unit testing is to go beyond these, and assess the execution of the edge cases. Using some material from my , some own work example would be -to test the validity of empty data/objects (where users wouldn't, generally, produce empty data instances, and this wouldn't be described in the manual page); -verify that an example object that is distributed with the package (and loaded with `load`), that was created from a distributed example files (for instance in `inst/extdata/`), remained valid and identical throughout the evolution of the classes and constructors; -other examples illustrate how unit tests can be build and refined against a set of functions specifications, i.e. expected outputs.
Similarly, code presented in examples illustrate exported and high level code, while unit tests might rather want to address specific and private functions, that in turn are called within the high level user space functions.
Another important point is that writing (or thinking about) unit tests often influences how one writes code, testable code, which is not reflected in writing code in the manual examples.
Also, although efficiency (in terms of time when checking the package) is not a concern, at least in a first instance, a side effect of basing unit test on examples, results in re-running the same code twice. For unit testing, it might be worth wile to extend code coverage to rarely used lines of code by tuning arguments accordingly.
In the vignette, the author clearly states that "The Goal is NOT Fully Automated Unit Test Creation". May be this could also be mentioned in the manuscript.
A minor suggestion would be to drop `covr` from the title, as it might mislead readers that the manuscript also addresses directly that package.
Also, I was wondering what happened if an example code chunk was not run (wrapped in `\dontrun`). Is it ignores by `exampletestr`?
To conclude, I think the manuscript would benefit greatly from a better introduction/discussion for newcomers. The second issue I have relates to the scope or validity of basing unit test on examples. While I do not expect `exampletestr` (or any other package) to come up with valid general unit tests generation recipe, I think it would be essential to recast and discuss unit testing into a broader scope.

Is sufficient information provided to allow interpretation of the expected output datasets and any results generated using the tool? Yes
Are the conclusions about the tool and its performance adequately supported by the findings presented in the article? Partly No competing interests were disclosed. I think it would be useful for the authors to provide a better rational for unit testing. In my opinion, the current version of the text is obvious to those that already know about and practice unit testing. However, the goal of this work is to appeal to the many that haven't adopted unit testing in their standard development yet. For those, it might be useful to provide more background and to emphasise the important of unit testing.
I have done this now in the introduction.

I have done this now in the introduction.
By focusing on code from examples, some of the most important unit tests are systematically missed out. Indeed, one generally documents a typical, default, best case application of the code/package in the documentation files. One could argue that the goal of unit testing is to go beyond these, and assess the execution of the edge cases. I have included this point in the discussion.
Writing (or thinking about) unit tests often influences how one writes code, testable code, which is not reflected in writing code in the manual examples. This is a good point. exampletestr is made for packages in which the author did not write tests as the package was being developed. In the manuscript I now try to make it clear that this isn't necessarily a good way to develop a tested package, but that it is the way that most packages are developed and hence exampletestr is a useful tool to begin testing these packages.
Although efficiency (in terms of time when checking the package) is not a concern, at least in a first instance, a side effect of basing unit test on examples, results in re-running the same code twice. This is unfortunate, however I think having a set of unit tests which affirm that your examples are working properly very useful and worth having in spite of this re-running (which is, as you eluded to, not a big concern most of the time).
For unit testing, it might be worth wile to extend code coverage to rarely used lines of code by tuning arguments accordingly. This is an important point. I think this is taken care of by the workflow that makes use of covr.
In the vignette, the author clearly states that "The Goal is NOT Fully Automated Unit Test Creation". May be this could also be mentioned in the manuscript. I decided not to copy this in explicitly because I think the discussion now makes clear the point that there is much more to unit testing than what exampletestr can do for you.
A minor suggestion would be to drop `covr` from the title, as it might mislead readers that the manuscript also addresses directly that package. I have done this. Please let me know if you think the new title could be improved.

Also, I was wondering what happened if an example code chunk was not run (wrapped iǹ \dontrun`). Is it ignores by `exampletestr`?
\dontrun{} blocks are ignored. I considered putting them into expect_error() statements but decided against it. I thought there was no good way to systematically deal with them. If you have a suggestion, please let me know.
To conclude, I think the manuscript would benefit greatly from a better introduction/discussion for newcomers. The second issue I have relates to the scope or validity of basing unit test on examples. While I do not expect `exampletestr` (or any other package) to come up with valid general unit tests generation recipe, I think it would be essential to recast and discuss unit testing into a broader scope. I have tried to address all of this. Thank you for your insightful review.