Systematics of Deliveries to Increase Postgraduate Courses Scientific Productivity

To assist researchers in the development of scientific studies and optimize the resources invested in engineering postgraduate programs, it is necessary instruments to support both teachers and students in carrying out the activities during the courses. Although there are proposals in the literature for structuring specific studies, there are gaps regarding the follow-up of these activities in a systematic and gradual manner. From an action research in a postgraduate course in engineering, in this study was developed, tested, evaluated and improved a tool to assist students in the development of subject-specific studies in postgraduate courses, identified as Systematics of Deliveries. The results show that the application of this tool can be useful not only in increasing the productivity of postgraduate courses, but also in the personal development of students as they follow the step-by-step the management of scientific studies.


Introduction
There is constant pressure on the academe to research and publish reports (Keen, 2007), especially in engineering postgraduate courses, in which the development of papers is often a criterion in performance evaluation.The embodiment of knowledge in such papers is considered as one way to overcome the limitations of research centers toward enhancing scientific productivity (Ponomariov & Boardman, 2010).Some studies have proposed ways to structure a scientific paper (Weiss & Newman, 2011), or even suggested the participation of writing coaches in this process (Baldwin & Chandler, 2002).However, there are limitations to the step-by-step elaboration of scientific papers in postgraduate courses to obtain the best possible results.
In developing courses in postgraduate programs, it is not enough to create demand for the advancement of knowledge, such as the need for sectorial studies, basic and applied research, or the production of knowledge as embodied in scientific papers.There is also a need for closer monitoring of these activities with the students in order to point out the mistakes and successes encountered along the way and thereby catalyze the role of academia in an innovation system involving the triple helix of university, industry and government (Etzkowitz & Leydesdorff, 2000).
Besides having the possibility of generating expertise in a given subject, the production of scientific papers can also enable researchers to explore unknown or tangent topics in their areas of expertise.Mode 2 of new knowledge production, as proposed by Gibbons et al. (1994) and discussed by Hessels and van Lente (2008), is characterized as, among others, presenting transdisciplinary elements and knowledge generated in the context of application, that is, the researcher identifies different areas of expertise by their reflection in real situations.However, it is noted that these activities are carried out in an almost individual way by postgraduate students, without a close accompaniment and orientation of the research activities to make knowledge tangible through scientific texts.
Therefore, the present study proposes a systematics for the monitoring and development of subject-specific studies in postgraduate courses.Previous results from the application of this systematics have shown a significant increase in the potential development of publications into a postgraduate course, as well as provided significant contributions to the postgraduate programs analyzed and the personal development of the students.

Methodological Procedures
This study was carried out in three distinct stages.The first one was related to the development and application of the proposed tool in an action research.The second one was the evaluation of this tool both in relation to its validity and its impact on the course activities, as presented in Table 1, and finally, in the third stage, improvements were made on the systematics proposed.
The course used as a reference was the Aeronautics Infrastructure.There were 37 students at the beginning of classes and 30 finished the course.Among the participants were students enrolled in master and doctoral courses, special students and non-regular enrollees in isolated courses, such as acting in companies and agencies in the airline industry.The large number of students enrolled in the course is mainly due to the fact that completion of this course is a requirement in the Postgraduate Program in Aeronautical Infrastructure Engineering (PG/EIA) at Aeronautics Institute of Technology (ITA), Brazil.This semester course was offered once a year and includes among its objects of evaluation the production of a scientific paper by the students.The score given for this paper is then integrated and calculated the students' final grade in the course.Historically, the production of such paper was considered mainly as an assessment tool.Among these works, one, two, or at most, three papers have been usually identified as having potential for publication.Such potential for publication is based not only on the score obtained in the evaluation, but mainly on the possible usefulness of the study in the academic community.By applying the proposed systematics, 13 papers were identified as having real potential for publication in journals or scientific events on transport management and engineering.
To evaluate the proposed Systematics of Deliveries, questionnaires were given to students of a postgraduate course and users to determine their views on the feasibility of this tool.The questionnaires were administered at the end of the delivery period and included 15 affirmative statements, to which the students could assign scores ranging from 1 (maximum disagreement with the statement) to 10 (maximum agreement).A total of 28 valid questionnaires were returned.
To analyze the validity of this systematics, concordance and relevance analyses were carried out with the students, and correlations were determined between the responses submitted to statistical analysis.
The analysis of the questionnaires considered the variance index (σ x 2 ) of responses for a given affirmative statement "x" and the range of 10 points (A) on the measurement scale of 1 to 10.These two values are used in Eq. 1, as proposed by James, Demaree and Gwolf (1984), to identify variations in the perceptions of students regarding the affirmative statements.
It was also necessary to obtain the means ( ) of the students' answers to the affirmative statements presented.These values, in addition to having the relevant level shown by students in relation to the affirmative analyzed, along with the standard deviation (S x ) of the average of responses, were used to identify the common perception of students related to the affirmative presented.
To obtain the level of agreement among the students regarding the affirmative statement "x", the concordance index was identified (C x ) from the answers given (James, Demaree, & Gwolf, 1993).This index, which was used to determine whether there was a consensus in the group regarding the statement presented, can be obtained by applying Eq. 2.
In Eq. 2, the value of C x ranges from 0 (maximum disagreement within the group regarding the affirmative statement presented) to 1 (maximum agreement).Farris et al. (2007) used this same method to evaluate the effectiveness of tools used in the development of new products.

Results
The results of evaluation the proposed Systematics of Deliveries are presented in Table 1, which shows the mean ( ), standard deviation (S x ) and concordance index (C x ) of the students' responses to the affirmative statements presented.
Table1.The students' responses The experience of the Collaborator in applying the Systematics was useful in developing the deliveries.9,1 1,0 0,9 The experience of the Collaborator in the field studied was useful to the research.7,3 3,0 0,0 The interaction mode (direct changes in the text of documents and e-mail messages) was adequate.9,3 1,2 0,8 Source: research data.
A detailed analysis of the data presented in Table 1 shows that, although there is low concordance among the respondents regarding the contribution of this Systematics to the development of the course under study (C x =0, 2), the average of the students' responses is high ( =7, 4).The same trend is observed regarding the contribution of the Systematics to the subject of Aeronautics Infrastructure (C x =0, 2; =6, 8); that is, on average, the proposed Systematics is useful to both the course and its theme.
Regarding the previous experience of students in the elaboration of scientific papers, heterogeneity was observed in the experiments (C x =0, 0).However, the average obtained was low ( =3, 4), indicating that the students had little experience in the preparation of such studies.On the other hand, the proposed Systematics helped increase the chances of developing and publishing papers ( =8, 5) for about half of the students (C x =0, 5).
Another relevant consideration was that the Systematics assisted the development of activities in the postgraduate program ( =8, 6; C x =0, 7).The results of the systematization may also be useful in the development of other studies or the preparing of dissertations and theses in the program.
One of the main contributions of this Systematics to the scientific development of the students was up with theories on the subject outside the classroom, as indicated by =8, 8, with a high level of agreement (C x =0, 8) among the students.
Even being able to consult with the practice on the subject outside the classroom ( =7, 0), something commonly requested between postgraduate students, it is noted that this was not a common agreement among the students (C x =0, 0).
In general, there was agreement regarding the suitability of the deliveries, with values ranging between 7, 6 and 8, 5.However, there was no agreement for all deliveries.Contributions were made to better distribution of time interval between deliveries and adjustments in the required formats, which have been improved in the proposal presented in this study.
Regarding the experience of Collaborator with the field studied, although there is no agreement in the responses and have a high average of 7, 3, there is one too high standard deviation, which shows certain limitations in knowledge of the Collaborator in the field studied in the paper.To improve this item, the topics could be restricted to the experiences of the Collaborator or the Professor in the course.
One of the aspects that received the most attention in this analysis and which is probably the main contribution of the systematics toward increasing productivity in postgraduate courses is the mode of interaction with the students, in which notes are inserted directly into Word-document deliveries.This item, which showed C x =0, 8, received the highest score among the evaluations ( =9, 3) because such close contact with the Collaborator or Professor helped the students in both drafting the text and planning the step-by-step development of the content.This close relationship with students is believed to be a key factor in optimizing the efforts invested in new knowledge production (Hessels & Lens, 2008).
The statistical tests showed significant correlations between the various affirmative statements presented (n=28).According to Table 2, there are several significant correlations between the affirmative statements analyzed (0.05 level (2-tailed)).However, considering the correlations at the 0.01 (2-tailed) in particular, significant correlations are observed between the contributions of the Systematics to the course under study and the potential of the students to learn more about the subject of their papers.The same trend applies to the experience of the Collaborator in the subject under study.
There are also significant correlations between the contributions of the Systematics to the postgraduate program and the students' increased chances of publishing papers.A significant correlation can also be observed between CP and deliveries D2 and D3, which deal with the theoretical bases and research methods.Thus, there is objective evidence that the proposed Systematics contributes not only to improving the postgraduate program, but also to increasing the chances of successful publication of scientific papers by students.
Although Table 1 indicates that the Systematics contributes considerably to increasing the students' chances of publishing papers ( =8, 5), Table 2 shows that there is no significant correlation between this variable and prior experience in writing papers ( =3, 4) (Pearson correlation=0, 302; Sig.(2-tailed)=0, 118).This means that, based on the operationalization of the Systematics, students have a good chance of publishing their studies despite having no previous experience in developing scientific papers.
Finally, the existence of significant correlations between all deliveries should be noted.Thus, although it is possible to make changes in the order of the deliveries, such changes should be made gradually, considering the step-by-step development of the study.

Proposed Systematics of Deliveries
After being evaluated and also carried out the necessary improvements, the Systematics of Deliveries is presented in order to optimize the efforts invested in postgraduate programs from the generation of specific knowledge by students on individual papers.
Commonly adopted in project and innovation management (Caetano & Amaral, 2011;Mesa, Molenaar, & Alarcón, 2016), the use of documents as partial deliveries toward a final project outcome generates results that are evaluated in the intermediate stages so that the end result is not put at risk.This proposed systematics also considers these partial deliveries to obtain the best final results possible.
Figure 1 shows the proposed Systematics of Deliveries in detail, including its description, the deliveries to be made, the approximate development time (in weeks) of these deliveries and some possible criteria for evaluating each of the deliveries in the composition of assessment notes in the course.
Ideas of topics and initial references.
Structure of the theoretical framework and initial concepts.
Methods and tools for data collection and analysis.
Analysis of results and final considerations.List of ideas with some of the course issues and also the relevance of identified articles.
Proximity of the items with the subject of study, logical structure of the text and theoretical basis consistency.
Feasibility of the study and relevance of procedures to be adopted.
Critical analysis of data, compared to previous studies and contributions of the results.
Relevance of the study both the theoretical and practical point of view, text structure and contribution to the theme.According to Figure 1, D1 consists mainly of a description of the main topics to be studied, along with the initial objectives of the study and the initial references on the subject.The list of ideas for the topics studied in the course, as well as the theoretical relevance of the references used in this delivery can be adopted as evaluation criteria.
In D2, the student presents the theoretical framework and the initial concepts of the topics discussed.This delivery considers the consistency of the theoretical basis used and the proximity of the concepts analyzed with the themes of the course.
D3 defines the methods and tools that will be used in the data collection and analysis, that evaluates the feasibility of performing such a study.This delivery requires prior knowledge of the instruments used in data collection and processing, such as survey methods and statistical analysis tools, among others.
D4 provides an analysis of the results in terms of both their theoretical and practical implications.In this delivery, the contributions of the study to the current state of knowledge are critically examined, and the limitations of the present study and the need for further research are described in the final considerations.
Finally, in D5 the student completes the paper by providing an introduction and a summary, making the final revisions, and coming up with a title that clearly indicates the contents of the document.D5 is assessed separately from the others as an individual delivery or as the result of the systematization of the other deliveries.
Figure 1 presents the deliveries as circles that vary depending on their individual possibility of improvement as the study is developed.For example, faced with limited access to primary or secondary data during D3, the student can adjust the theoretical basis in D2 according to a possible alternative idea generated in D1, thereby optimizing the study.
In the case studied herein, the completion of the deliveries by the students took about 18 weeks in total from D1 to D5, or the equivalent of about 4 months of classes.This period corresponds to a semester class, which is commonly adopted in postgraduate courses.
The monitoring and evaluation of the deliveries involved several interactions with the students during the semester, which also required a considerable amount of dedicated time.The total number of interactions (i) and time commitment (t), in minutes, used in the analysis of the deliveries were as follows: i=48 and t=580 for D1, i=46 and t=920 for D2, i=38 and t=450 for D3, i=30 and t=390 for D4, and i=13 and t=500 for D5.A total of 175 interactions were carried out by email, beyond the required weekly meetings for the course.About 2.840 minutes or almost 50 hours of monitoring and analysis of deliveries were also done with the students.
Note that the delivery to which a greater amount of time was dedicated was D2, in which the main concepts associated with the studies and their contributions to the state of knowledge were analyzed.D1 also required a significant amount of time for analysis because it included 48 different deliveries in various improved versions.
To a lesser extent, more time was dedicated to D5, in which final adjustments were made on papers intended for submission to events and journals.
Of the 37 students who started this course, only 29 completed the deliveries.The others dropped out of the course, because there is the possibility of participation of non-regular students, and one student did not reach the D5.Among the students who submitted papers with potential for publication, the average D5 score was 9.4, and the others had an average score that was more than 2 points lower, at 7.3.
In postgraduate programs, the paper is usually part of the course evaluation, as in this case, or a dissertation or final thesis has to be submitted.In some cases, both are required.The use of these results in the evaluation of the student can be performed from the participation percentages on Final Grade (FG) in the course, as shown in Eq. 3, where the average ratings of deliveries D1, D2, D3 and D4, added to note the final paper (D5), corresponding to 50% of the final grade.The remaining 50% can be distributed among other forms of assessment, such as tests and seminars.It should be noted that the monitoring of deliveries must be done by a Collaborator, as this case, or the Professor of the course.Because the activities of preparing and administering the course consumes a significant amount of time for the Professor, the participation of such Collaborator in this process may be crucial.

Conclusion
A Systematics of Deliveries aimed at optimizing the efforts invested in postgraduate courses is proposed in this study.After carrying out the necessary reviews and improvements, significant contributions of the Systematics to improving the students' capabilities and increasing productivity in the postgraduate program were identified.The proposed Systematics can be adjusted according to the realities of the course in which it is applied.
Despite the demonstrated advantages of the proposed tool, the monitoring students in the development of their studies is crucial.The role of a Collaborator, as a qualified Professor assistant, is essential in coaching the students in each step of the process as they gradually form the body of the research and prepare the final paper.
There was a significant increase in the number of papers with potential for publication.However, future research could focus on identifying the reasons why other papers do not pass the evaluation, so that the efforts invested in the development of these papers can be further optimized.

Figure 1 .
Figure 1.Schema of the proposed Systematics of Deliveries This Systematics enables me up with theories on the subject outside the classroom.This Systematics enables me up with practices on the subject outside the classroom.

Table 2 .
Table 2 presents only the highest correlations identified in the analysis, especially those that are significant at 0.05 and 0.01 level.Items with no significant correlations are not included in the table.Main correlations between the statements analyzed