THE REVIEWING PROCESS FOR ISPRS EVENTS

Following the first initiatives taken by the International Programme Committee of the XXIII ISPRS Congress in Prague (Czech Republic) in 2016, modifications of the reviewing process of ISPRS events were further considered during the years 2017 and 2018. This evolution first targets to better fit such a process to the currents requirements and expectations of the ISPRS community. Secondly, it aims to provide unified guidelines for the different steps of the process. Under the aegis of the 2020 Congress Director and ISAC (International Science Advisory Committee) chair, several discussions were held in-between September 2017 and June 2018 with ISAC members, Technical Commission Presidents (TCP), council members, 2016 and 2020 Congress Programme Chairs. This document serves as a unique transparent basis that applies for all kinds of ISPRS events (from Congress and Geospatial Week to smaller workshops), and all categories of people that are bound to be involved in the evaluation process of scientific contributions (authors, reviewers, TCPs, . . . ). It also specifies the evaluation criteria for the works submitted to ISPRS events, both for full papers and abstracts. Subsequently, it helps authors to improve the content and shape of their contributions. Eventually, this paper is targeted to help new chairs to smoothly prepare their future event. The following guidelines were first adopted for the 2018 Technical Commission Symposia.


INTRODUCTION
The International Society for Photogrammetry and Remote Sensing (ISPRS) organizes and supports various scientific events through the Congress Director (the Congress, every 4 years), the Technical Commission Presidents (the symposia, every 4 years), and other Event Chairs (e.g., the Geospatial Week, every 2 years, and smaller conferences and workshops).So far, the reviewing process (important dates, assignment to reviewers, evaluation form, final decision. . . ) varies with respect to the event: size, topics, community.In particular, it has been noticed that the evaluation criteria of a given event may favour either method-based papers or application-based papers.Therefore, there is no global coherency, except the final publication of the accepted papers in the ISPRS Annals and Archives (See Section 2 and (ISPRS, 2018d)).
The International Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences (ISPRS Annals) contain double blind peer reviewed scientific contributions of ISPRS Congresses, Symposia and a number of conferences and workshops (ISPRS, 2018b).The series was newly established in 2012.The Annals are open access publications, published under the Creative Common Attribution 3.0 License.The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences (ISPRS Archives) is the series of peer-reviewed proceedings published by ISPRS since 1908(ISPRS, 2018c).Since 1999 and 2011, the Archives are open access and published under the Creative Common Attribution 3.0 License, respectively.The Annals and the Archives are listed in the ISI Conference Proceedings Citation Index (CPCI) of the Web of Science, SCOPUS, and DOAJ.
In 2016, the International Programme Committee of the XXIII rd Congress in Prague (Czech Republic) established, during the preparation of the congress, guidelines for the reviewing process (IS-PRS, 2018a).In 2017, the ISPRS Council has decided to refine such guidelines and propose them as standards for all forthcoming ISPRS events.In addition, under the aegis of the 2020 Congress Director and ISAC (International Science Advisory Committee) chair, standardized evaluation criteria and their respective description have been set up.They stem from discussions with ISAC members, 2016-2020 Technical Commission Presidents (TCP), council members, 2016 and 2020 Congress Programme Chairs.Several iterations existed between May 2017 and June 2018.This paper documents the newly established reviewing guidelines for scientific contributions submitted to any kind of ISPRS events (from Congress and Geospatial Week to smaller workshops).It describes the evaluation sequence for all categories of people that are involved in the process: authors, reviewers, TCPs, . . . .It also specifies the evaluation criteria for the papers submitted to ISPRS events.It applies both for full papers and abstracts.Subsequently, it helps authors to improve the content and shape of their contributions.This paper is eventually targeted to help new chairs to smoothly prepare their future event.The following guidelines were first adopted for the 2018 Technical Commission Symposia.The document is structured as follows.Information about paper submission is first provided, both for full papers and abstracts (Section 2), followed by a comprehensive description about the reviewing process (Section 3).Then, guidelines for Programme Chairs/Area Chairs and reviewers are detailed in Sections 4 and 5, respectively.

SUBMISSION PROCEDURE
The aim of this section is two-fold.First, it presents the ISPRS framework for the evaluation process (publication, deadlines, criteria).Secondly, it explains what is expected from an author during the submission and evaluation processes.The new criteria (and their description provided in the review form of each ISPRS event) are targeted to be applicable both to "methods" and "application" papers, and to avoid favouring one of these two categories.We provide these criteria for the sake of transparency and in order to help the authors better shaping their contribution prior to submission.Templates for the papers to be submitted can be found on the ISPRS website (ISPRS Congress Programme Committee, 2016).The details of the Technical Commissions and the Working Groups are provided in the following web page: http://www2.isprs.org/commissions.html.

Full papers
2.1.1Explanation.A submitted full paper will undergo a double blind review.The paper should be anonymous.It is recommended to mention in which Working Group the authors believe it best fits into.Based on the evaluations provided by all persons involved in the decision process (see Section 3.1), authors will be notified about acceptance or rejection.Accepted papers will be published either in the ISPRS Annals or in the ISPRS Archives, depending on their quality (very good or correct, respectively).The decision is taken by the chair(s) of the scientific committee of the event (see below).This committee also decides whether the contribution, if accepted, is presented in an oral or an interactive session.More details are provided in Section 3.1.They will also be specified on the website of the corresponding event.
2.1.2Tentative deadlines.The following schedule is provided so as to let the authors know approximately the time range of each step of the submission/reviewing process.Each Event Chair can modify this schedule, depending on the date of the event and potentially temporal conflict with other ISPRS events and non-ISPRS conferences with close topics and/or similar communities.
• Full paper submission: 4-5 months before the event, depending on the expected total number of submissions; • Notification of acceptance: 2 months later; • Camera-ready paper due: 1 month later; • Early-bird registration: close to the camera-ready deadline.

Evaluation criteria.
Each author should take into account that reviewers will look for answers to the questions inserted in the review form.The evaluation sheet is composed on two main parts: (1) an overall recommendation score (40-50% of the total, depending on the event), and (2) a list of criteria (50-60% of the total), for which each reviewer should provide a score and a detailed evaluation (see Section 5).These questions are provided in Section 2.3.The advantage of such a list is that all criteria are of similar importance.They should be weighted equally so as to provide a final mark that is consistent whatever the scope of the paper (method-driven or application-driven).
The overall recommendation (and the associated confidence score) and the technical evaluation of all the reviewers are taken into account so as to first decide for acceptance/rejection, and then, for accepted papers, for oral or interactive presentation.More details are provided in Section 3.1.

Policies
• By submitting a paper, the authors agree to the review process.
• The authors should avoid providing information that may identify themselves in the main body of the paper, as well as in the acknowledgements (e.g., co-workers and grant IDs).
If a different paper of the authors is cited, anonymity should be preserved.
• By submitting a paper, the authors acknowledge that it has not been previously submitted in a similar shape in any peer reviewed conference.Besides, no similar paper can be submitted to another conference during the review period.Violation of these conditions leads to rejection.The submission to non peer reviewed public repositories like arXiv is nevertheless allowed.Duplication publications and plagiarism are major concerns for the ISPRS community (Lichti, 2018): the ISPRS journals and most of the ISPRS events adopted plagiarism detection softwares.
-The definition of a "similar shape" is based on the IS-PRS policy on pre-prints in public repositories ((IS-PRS, 2018d), Sections 3 and 4) and is also subject to the assessment of the chair(s) of the scientific committee.-Any submission should not have strong overlap with prior papers and other concurrent submissions.-The authors can contact the chair(s) of the scientific committee for borderline cases, prior to submission.
• By submitting a full paper, the authors agree that if the paper is accepted, at least one of the authors will register for the event (before the early-bird deadline), pay the registration fee, and present the paper there.Otherwise, the paper will not be included in the ISPRS Annals or Archives.

Abstracts
2.2.1 Explanation.Submitted abstract is reviewed.The paper can be anonymous, depending on the policy of the event.It is recommended to mention in which ISPRS Working Group the authors believe it best fits into.Based on the evaluations provided by all persons involved in the decision process, authors will be notified about acceptance or rejection.If accepted, final paper will be published in the ISPRS Archives.The scientific committee also decides whether the contribution falls into an oral or an interactive session (see Section 3.1).One should note that an increasing number of Event and Programme Chairs favour full paper submissions: they use the rule that papers accepted on an abstract basis will only be presented in interactive sessions.This rule is set up independently for each event.

Tentative deadlines.
The following schedule is provided so as to let the authors know approximately the time range of each step of the submission/reviewing process.Similarly to events based on full paper submission, the following schedule can be modified by the Event Chair(s).
• Abstract submission: 3-4 months before the event depending on the expected total number of submissions; • Notification of acceptance: 1 month later; • Full paper due: 1/1.5 month later; • Early-bird registration: close to the full paper deadline.

Evaluation criteria and policies
Each author should take into account that reviewers will look for answers to the questions inserted in the review form.The overall framework is the same as for full papers (details in Section 2.3): an overall recommendation accompanied with a list of equally weighted questions.However, the questions are specific to abstract-based submissions.
By submitting a paper, the authors agree to the review process and to upload a full paper if the paper is accepted.At least one of the authors will register for the event (before the early-bird deadline), pay the registration fee and present the paper there.Otherwise, the paper will not be included in the ISPRS Archives.

Reviewer questions
Five main questions compose a review form for ISPRS events.The criteria slightly differ depending whether a full paper or an abstract is submitted.
• Innovation: -Full paper: Does the paper contain original/innovative work (on theory and/or methods and/or applications)?Has related work been properly credited?
-Abstract: Does the abstract indicate original/innovative work (on theory and/or methods and/or applications)?Is there any mention of existing approaches?
• Scientific formulation: -Full paper: Are the hypothesis clearly stated?Are the research questions correctly formulated?Are the employed methods appropriate?Are equations / methodological framework correctly presented / justified?
-Abstract: Is the abstract informative?Does it contain information about the hypotheses, the research questions, the methods employed and their relevance?
• Experiments and validation: -Full paper: Do the authors properly reflect on the achieved performance of the proposed methods or procedures?Are facts and interpretation duly separated?Are results assessed in a qualitative and quantitative way?Are the interpretations and conclusions justified by the data?
-Abstract: Do the authors properly reflect on the expected performance of the proposed methods or procedures?
• Relevance: Does the subject fit within the call for papers?Do the results have a practical relevance?Will results impact other research?
• Presentation: -Full paper: Is the paper well written and well structured?Are figures well readable?Are the references complete?Is the paper formatted according to the author guidelines?
-Abstract: Is the abstract well written?Do the authors provide some references and an illustration of their work?Is the abstract formatted according to the author guidelines?

Number of submissions per author
There is no limit for the number of submissions per submitter and per co-author.However, there is a limit on the registration level: one presented paper should correspond to one registration fee.If there is no corresponding registration for a given paper, it will not be included in the ISPRS Annals or in the ISPRS Archives.

Issues
For all issues, including a possible conflict of interest during the reviewing process, the rules of the ISPRS Orange Book apply. (http://www.isprs.org/documents/orangebook/app4.aspx).

Organization
The evaluation process is traditionally organised with a 4-level structure, under the control of the scientific responsible of the event.The latter one is generally either the Programme Chair(s) or directly the Event Chair(s).Such an organization has been adopted by many computer science communities and scientific societies in the last two decades: e.g., image processing, computer vision, computer graphics, machine learning, IEEE, ACM, EURASIP (Hinckley, 2015, CVPR, 2018, MICCAI, 2018, Shah et al., 2017, NIPS, 2018).The four levels are: (i) The authors, who submit a contribution with a topic that fits to one or several ISPRS working groups (WG, see ISPRS website for more details).
(ii) The reviewers, that evaluate these contributions following a list of criteria.This list is now the same whatever the event (see Section 2.3).
(iii) The Area Chairs (AC), that select qualified reviewers and take preliminary acceptance/rejection decisions for papers in which they are experts.AC are often the officers of the WG related to the submission1 .They also carefully read the papers and animate a discussion phase after a first round of review, in addition to the review collection process (see (CVPR, 2014) for more details).For events with a significant number of papers, it may be necessary to have more AC than current WG officers.
(iv) Eventually, the Programme Chair (PC), that first assigns papers to the Area Chairs and ensures a homogeneous repartition of papers between AC.Then, the PC checks whether papers are consistently, fairly, and timely evaluated.The PC ensures the harmonization between all AC.The PC is responsible in building the scientific programme, and, most of the time, takes final decision, based on AC recommendations.Details are provided in Section 4.

Event specificities
The generic "author←reviewers↔area chairs↔programme chair" sequence is modified as follows depending on the ISPRS event: • Congress: Technical Commission Presidents act as Programme Chairs for their own commission.They assign the papers to AC, interact with them for consolidating the reviews, and take the acceptance/rejection decision.AC are selected among WG officers and external experts if required.Subsequently, the Congress Programme Chair only builds the final programme (oral/interactive sessions, long/short presentation), both in accordance with the 5 TCPs and by taking into account the other existing sessions (special and thematic sessions).
• Symposia: TCP act both as Event and Programme Chairs.
Similarly to congresses, the reviewer list is mainly composed of WG officers, complemented with additional experts.
• Geospatial Week (GSW): the various events are run independently.Therefore, the Event Chairs are their own Programme Chairs and take both acceptance / rejection and oral / interaction presentation decisions.The area chair level is most of the time maintained only for large events where AC perform both review and review consolidation (e.g., Laserscanning, Silvilaser, UAV-g).Each Event Chair selects the AC.Similarly to congresses, the GSW director (or the GSW Programme Chair) ensures a smooth coordination and scheduling between events but do not intervene in the review process.
• Conferences and workshops (out of the GSW): events chairs act as programme chairs.The area chair level is most of the time removed.
Event/Programme Chairs can additional decide to set up a bidding phase for their event.In such a case, experts have access to either the full list of papers or the list of papers that fall into their domain.Then, they can label their willingness to review these papers.The taxonomy can be for instance: "Yes/Eager to", "Willing", "In a pinch", "No".The final paper assignation is partly based on such a labelling, in a global optimization procedure.'Similarly, a rebuttal phase can be established.Reviews are released to the authors that have the possibility to answer to reviewers' comments in a concise way and in a short period of time (often 1 week).The rebuttal letter is included in the subsequent discussion phase between reviewers and area chairs and, consequently, in the final decision process.

Workflow
We distinguish 2 cases: large and small events.Large events involve Area Chairs in-between Programme Chairs and reviewers (see above).These events are likely to correspond to ISPRS congresses and symposia.GSWs fall into the second category since each event hosted within the GSW framework is independent.
Case 1: Large events (involvement of area chairs) • PC assign papers to an AC.The AC are often Working Group officers, and the authors select the WG their paper fall into.Thus, this step often only consists in assigning a paper to a short list of 3-5 AC.
• AC are responsible for fast checking the papers, assigning reviewers and collecting the reviews for papers assigned to them, and for making a preliminary decision.The AC do not make reviews themselves: they make their decision based on the collected reviews, and their own opinion on the quality of a paper.In this decision making, the AC officers can ignore reviews of a poor quality (e.g., missing motivation for negative assessments, sparse comments).They are asked to take position if reviewers expressed contradicting views on the quality of a paper.In such a case, they can even ask for emergency reviews.
• AC should report any conflict of interest as soon as possible, so that PC can transfer papers to another AC.
• PC check the preliminary decisions of the AC and make the final decision.PC can involve another AC for decision taking if necessary.
• PC decide whether the paper goes to an oral or an interactive session, based on AC recommendations.When cameraready papers are submitted, PC check whether papers are layed out in line with the ISPRS template (Section 2).
• All accepted papers are checked and formatted by Copernicus for referencing and publication on ISPRS website.Copernicus GmbH is the proceeding publisher for ISPRS Annals and Archives.
Case 2: Small events • PC fast check the papers, assign reviewers, collect the reviews, and make a decision.The PC do not make reviews themselves: they make their decision based on the collected reviews and their own opinion on the quality of a paper.In this decision making, the PC officers can ignore reviews of a poor quality (e.g.missing motivation for negative assessments), and take a decision even if reviewers expressed contradicting views on the quality of a paper.Emergency reviews can be requested.
• If there are multiple PC for the same event, any paper handled by a PC with a conflict of interest should be transferred to another PC.
• If a single PC exists, coupled with a relatively high number of papers, then a limited number of AC (e.g., <10) can be selected by the PC.
• When camera-ready papers are submitted, PC check whether papers are layed out in line with the ISPRS template.
• PC send all accepted papers to Copernicus for referencing and publication on ISPRS website.

Initial fast checks
Some authors may have selected the wrong session / Working Group / Technical Commission.Therefore, AC or PC check if the paper topic is appropriate.If not, it should be reassigned by (one of) the PC or the Event Chair.If the PC/AC considers the paper inappropriate for a specific topic/TC, they assign the paper to another TC/AC.It may happens for congresses and symposia.For the specific case of the GSW, transfer between events can be proposed to the authors for a more adequate paper handling.PC/AC also check whether the paper is anonymous.If not, they ask the author to upload an anonymised version.PC/AC eventually briefly check whether the paper was edited based on the ISPRS template and are that it is readable (encoding, corrupted PDF).If the paper is not readable or obviously not correctly formatted (e.g.one-column format, wrong reference style, over-length), they ask the author to upload a corrected version in good time.Papers presenting a conflict of interest are re-assigned to another PC/AC.

Reviewing process
The PC/AC decide the number of reviewers that should be invited per paper.This number should be constant per event (2-4) but cannot be guaranteed (due to missing reviews in particular).This number depends on the length of the papers and the size of the programme committee of the event (Price and Flach, 2017).PC/AC then assign reviewers for every paper.PC/AC do not review the papers of their topic themselves.They are supposed to read the papers on which they need to make a preliminary decision.If the number of papers in a topic is too large (e.g., more than 10), the AC can ask the PC to appoint additional area chairs and reassign some of the papers to them.Most of the time, PC/AC decide themselves who are the most adequate reviewers for each paper.The assignment process can be made easier with a bidding procedure (Section 3.2), and more recently with automatic tools such as the Toronto Paper Matching system (Charlin and Zemel, 2013).Such solutions are not fully satisfactory since it remains difficult to satisfy fairness, expertise, and interest constraints for everybody at the same time (Karimzadehgan andZhai, 2009, Long et al., 2013).PC/AC monitor the reviewing progress and timely invite additional reviewers if reviews are expected to be late or unavailable.If a submitted review has a very low quality, PC/AC can invite additional reviewers.AC can be reviewer for papers of other topics, but should consider whether they can complete the reviews in time along with the work for their own WG.

Decisions
The AC discuss each paper and make their decision based on the collected reviews and their own opinion on the quality of a paper.In this decision making, AC can ignore reviews of a poor quality and can be asked to take position if reviewers expressed contradicting views on the quality of a paper.Possible decisions are Accept for Annals (limited to the best rated full papers), Accept for Archives (full papers and abstracts), and Reject.Whether an accepted paper will be presented in an oral or poster session is decided later (decision of the Programme Chair).However, AC can give their point of view.For Case 1 only, PC check the preliminary decisions of the AC and make the final decisions.To avoid any conflict of interest, final decisions on papers with PC (co-)authorship will be made by other PC if possible.Decisions are eventually communicated to the authors.

Deadlines
The assignment to AC is performed within 1 week after the submission deadline.Initial check is done immediately after the assignment (Case 1) or after submission deadline (Case 2).For Case 2, it can even be performed on-the-fly during the submission period.Assignment to reviewers should be done 1 week later.It can be extended to 2 weeks if a bidding procedure is decided.Full paper and abstract reviews are approximately due 4-6 weeks and 2-3 weeks later, respectively, depending on the time allocated by the PC.Preliminary decision by AC and potentially additional AC involved (Case 1) is provided 1 week later.Final decision happens 1 week later, immediately followed by the notification of acceptance.

Policies
Reviewers should preserve their anonymity., 2018)).Pros and cons exist; the impact of such an innovation compared to the standard double blind peer review has not been assessed so far (Soergel et al., 2013, Walker andRocha da Silva, 2015).

Full papers
In addition to rating the papers according to the five criteria (see Section 2.3), reviewers are asked to explain their evaluation in a detailed and clear manner and justify the above scores.The comments should be specific since discussions are more helpful than scores, both for authors and for the final decision.We recommend to point out strengths and weaknesses of the submitted contribution.In addition, reviewers should provide suggestions for improvement for the camera-ready paper and use an objective and constructive writing style.References to significant missing papers should be provided, excluding as much as possible papers from reviewers' group and their close collaborators.
Minor comments should also be provided (typos, erroneous sentences, additional references to be added, problems with the paper format) since authors are bound not to proof read their paper before sending their camera-ready version (due to limited time frame).Innovation and stimulation are core elements in an ISPRS event.
Acceptance or rejection decisions should not be advocated solely by the performance of the proposed methods.It is not mandatory for the authors to propose methods that exceed the accuracies of the state-of-the-art or that have been tested on specific benchmark datasets.Nevertheless, papers should be technically correct and should propose a contribution to their field.

Abstracts
Once accepted, abstracts will not be reviewed again.Therefore, the degree of maturity of the work should be clearly assessed in the reviewer score sheet.This is absolutely necessary so as to build a consistent programme, especially if some abstract-based works are targeted to be included in oral sessions.Suggestions on improvements, method comparison, references to existing works should be provided with the review so that the authors can take them into account.Finally, the reviewers should note that at the stage of the submission experimental results may not be available and fully analysed.Therefore, the reviewers should not focus their evaluation on the "Experiments and validation" criterion.

CONCLUSION
In this paper, we comprehensively detailed the submission and review process for ISPRS events.It applies for full papers and abstract submissions, for method-driven and application-driven contributions.The benefit is two-fold: a transparent and unified description for (i) authors preparing their manuscript and (ii) scientists involved in all steps of the review process, up to the chair of the event.We described the overall framework and explained how it can be implemented for specific cases: the chair(s) of the event can further tuned such a process so as to better fit to specific needs.We proposed a refined set of criteria for the evaluation of papers, which will be the basis for all forthcoming ISPRS events.
The Conference Management System in which the authors uploaded their contribution already ensures that reviews are blind.The authors should avoid providing any hint about who they are: names, affiliations, reference to papers and projects, acknowledgement should be omitted.Reviewers have the responsibility to keep the submissions confidential, including papers rejected from the event it has been submitted to.If reviewers are assigned a paper with a possible conflict of interest, they should return the paper immediately to their Area Chair or Programme Chair.Details about the review procedure and the ISPRS policy on preprints in public repositories are available at the following link: http://www.isprs.org/documents/orangebook/app4.aspx.Citations to these papers are not required and such reference omission is not ground for rejection.Reviews are available to other reviewers, AC, PC, and the authors of the papers.Only few conferences have decided to adopt the open-review strategy: the comments are publicly available, and the authors can provide continuous feedback during the evaluation time frame (see for instance (ICLR