ABSTRACT

Objective:

To carry out the National Immunization Program Information System (SIPNI) accessability assessment (EA), considering the description of the intervention, the elaboration of the theoretical-logical model and the identification of the evaluative questions.

Method:

A single case study with a qualitative approach, based on the system of seven elements proposed by Thurston and Ramaliu (2005). The SIPNI and the context of analysis, the vaccination rooms, were defined as the case.

Results:

The SIPNI description, based on documents and scientific evidence, made it possible to understand its operationalization, the main events that characterize it. The theoretical-logical model visually and systematically configured the SIPNI organization with details of the inputs, activities, products and expected effects. The evaluative questions were evaluated and validated to evaluate the SIPNI.

Final considerations:

The SIPNI is evaluable and presented a favorable scenario for its development. EA is an important strategy for planning subsequent evaluations, maximizing their potentials.

Descriptors:
Immunization; Immunization Programs; Health Information System; Evaluation Studies; Nursing

RESUMEN

Objetivo:

Realizar Estudio de Evaluación (EA) del Sistema de Información del Programa Nacional de Inmunización (SIPNI), considerando la descripción de la intervención, la elaboración del modelo teórico-lógico y la identificación de las preguntas evaluativas.

Método:

Estudio de caso único, con enfoque cualitativo, basado en el sistema de los siete elementos propuestos por Thurston y Ramaliu (2005). Se definió como caso el SIPNI y el contexto de análisis, las salas de vacunación.

Resultados:

La descripción del SIPNI, fundamentada en documentos y evidencias científicas, posibilitó comprender su operacionalización, los principales acontecimientos que lo caracterizan. El modelo teórico-lógico configuró de forma visual y sistemática la organización del SIPNI con detalle de los insumos, actividades, productos y efectos esperados. Se establecieron y validaron las preguntas de evaluación para evaluar el SIPNI.

Consideraciones finales:

El SIPNI es evaluable y presenta un escenario favorable para su desarrollo. El EA es una estrategia importante para la planificación de evaluaciones posteriores, maximizando sus potenciales.

Descriptores:
Inmunización; Programas de Inmunización; Sistema de Información en Salud; Estudios de Evaluación; Enfermería

RESUMO

Objetivo:

Realizar Estudo de Avaliabilidade (EA) do Sistema de Informação do Programa Nacional de Imunização (SIPNI), considerando a descrição da intervenção, a elaboração do modelo teórico-lógico e a identificação das perguntas avaliativas.

Método:

Estudo de caso único, com abordagem qualitativa, baseado no sistema dos sete elementos propostos por Thurston e Ramaliu (2005). Definiu-se como caso o SIPNI e o contexto de análise, as salas de vacinação.

Resultados:

A descrição do SIPNI, fundamentada em documentos e evidências científicas, possibilitou compreender sua operacionalização, os principais acontecimentos que o caracterizam. O modelo teórico-lógico configurou de forma visual e sistemática a organização do SIPNI com detalhamento dos insumos, atividades, produtos e efeitos esperados. Foram estabelecidas e validadas as perguntas avaliativas para avaliar o SIPNI.

Considerações finais:

O SIPNI é avaliável e apresentou cenário favorável para o seu desenvolvimento. O EA é uma estratégia importante para o planejamento de avaliações subsequentes, maximizando seus potenciais.

Descritores:
Imunização; Programas de Imunização; Sistema de Informação em Saúde; Estudos de Avaliação; Enfermagem

INTRODUCTION

The Health Information Systems (HIS) consists of the joining of interrelated components that collect, process, store, disseminate information and guarantee the production of knowledge that is processed according to interests in terms of its use and its effects on practices of everyday life in health⁽¹1 Pinheiro ALS, Andrade KTS, Silva DO, Zacharias FCM, Gomide MFS, Pinto IC. Health Management: the use of information systems and knowledge sharing for the decision making process. Texto Contexto Enferm [Internet]. 2016 [cited 2017 May 26];25(3):e3440015. Available from: http://www.scielo.br/pdf/tce/v25n3/0104-0707-tce-25-03-3440015.pdf
http://www.scielo.br/pdf/tce/v25n3/0104-... ⁾. The SIS conform as management tools with great potential for planning and decision making in the organization of health services⁽²2 Cavalcante RB, Kerr Pinheiro MM, Guimarães EAA, Miranda RM. Panorama de definição e implementação da Política Nacional de Informação e Informática em Saúde. Cad Saúde Pública [Internet]. 2015 [cited 2017 May 27]; 31(5):960-70. Available from: http://www.scielosp.org/pdf/csp/v31n5/0102-311X-csp-31-5-0960.pdf
http://www.scielosp.org/pdf/csp/v31n5/01... ⁾.

In Brazil, the development and implementation of several IIS has been promoted, among them the National Immunization Program Information System (SIPNI), implemented with the purpose of collecting data on vaccination activities in order to generate individualized information from of the local authority to assist the decisions and actions in its management⁽³3 Brasil. Ministério da Saúde. Secretaria de Vigilância em Saúde. Departamento de Vigilância das Doenças Transmissíveis. Manual de vigilância epidemiológica de eventos adversos pós-vacinação. 3. ed. Brasília: Ministério da Saúde; 2014. 250 p.^-44 Luhm KR, Waldman EA. Sistemas informatizados de registro de imunização: uma revisão com enfoque na saúde infantil. Epidemiol Serv Saúde [Internet]. 2009 [cited 2017 Jan 14]; 18(1):65-78. Available from: http://scielo.iec.pa.gov.br/pdf/ess/v18n1/v18n1a07.pdf
http://scielo.iec.pa.gov.br/pdf/ess/v18n... ⁾. The implementation of such technologies is expected to improve the quality of information for the planning and evaluation of immunization actions in a municipal context⁽⁵5 Sato APS. National immunization program: computerized system as a tool for new challenges. Rev Saúde Pública [Internet]. 2015 [cited 2016 Dec 10];49:39. Available from: http://www.scielo.br/pdf/rsp/v49/0034-8910-rsp-S0034-89102015049005925.pdf
http://www.scielo.br/pdf/rsp/v49/0034-89... ⁾.

The implementation of the municipal SIPNI aims to manage the quality of the information of the vaccinated people during the life of the patient, the notification of the scheduled and delayed vaccines, the active surveillance of the adverse events following vaccination (AEFV), the control of the immunobiological validity and the lot of the vaccines in use and in the stock⁽⁶6 Brasil. Ministério da Saúde. Datasus. SI - PNI - Sistema de Informações do Programa Nacional de Imunizações [Internet]. 2017 [cited 2017 Jan 28]. Available from: http://datasus.saude.gov.br/sistemas-e-aplicativos/epidemiologicos/si-pni
http://datasus.saude.gov.br/sistemas-e-a... ⁾. Its use will allow the identification of pockets with low vaccination coverage, epidemiological surveillance of immunopreventable diseases, active surveillance of AEFV and, if related to other databases, can leverage vaccine impact assessment studies⁽⁴4 Luhm KR, Waldman EA. Sistemas informatizados de registro de imunização: uma revisão com enfoque na saúde infantil. Epidemiol Serv Saúde [Internet]. 2009 [cited 2017 Jan 14]; 18(1):65-78. Available from: http://scielo.iec.pa.gov.br/pdf/ess/v18n1/v18n1a07.pdf
http://scielo.iec.pa.gov.br/pdf/ess/v18n... ^-55 Sato APS. National immunization program: computerized system as a tool for new challenges. Rev Saúde Pública [Internet]. 2015 [cited 2016 Dec 10];49:39. Available from: http://www.scielo.br/pdf/rsp/v49/0034-8910-rsp-S0034-89102015049005925.pdf
http://www.scielo.br/pdf/rsp/v49/0034-89... ^,77 Community Preventive Services Task Force. CPSTF. Recommendation for use of immunization information systems to increase vaccination rates. J Public Health Manag Pract [Internet]. 2014 [cited 2017 Jul 20];21(3):249-52. Available from: http://journals.lww.com/jphmp/Fulltext/2015/05000/Recommendation_for_Use_of_Immunization_Information.3.aspx
http://journals.lww.com/jphmp/Fulltext/2... ⁾.

However, it is assumed that there are differences in the process of SIPNI implementation between municipalities. These differences were influenced by the non-implementation of a health informatics and information policy aimed at organizing the processes of production, use and dissemination of information and the insufficient training of human resources for information management⁽²2 Cavalcante RB, Kerr Pinheiro MM, Guimarães EAA, Miranda RM. Panorama de definição e implementação da Política Nacional de Informação e Informática em Saúde. Cad Saúde Pública [Internet]. 2015 [cited 2017 May 27]; 31(5):960-70. Available from: http://www.scielosp.org/pdf/csp/v31n5/0102-311X-csp-31-5-0960.pdf
http://www.scielosp.org/pdf/csp/v31n5/01... ⁾. In addition, the decentralization of a technological innovation implies structural and process changes in the day-to-day life of health professionals and users of services, and therefore it is a complex process that will determine, on the one hand, the effective adoption of new technology or in its rejection⁽⁸8 Holmes ES, Santos SR, Almeida AF, Oliveira JHD, Carvalho GDA, Fonsêca LCT, et al. Health information systems in the decision-making process in primary care. Int Arch Med [Internet]. 2016 [cited 2017 Feb 10];9. Available from: http://imed.pub/ojs/index.php/iam/article/view/1380/1116
http://imed.pub/ojs/index.php/iam/articl... ⁾.

The SIPNI is in an advanced stage of implementation in Brazil, but still faces challenges and obstacles related to the training of professionals, to the field of technology, to maintaining the system and guaranteeing the confidentiality of information⁽⁵5 Sato APS. National immunization program: computerized system as a tool for new challenges. Rev Saúde Pública [Internet]. 2015 [cited 2016 Dec 10];49:39. Available from: http://www.scielo.br/pdf/rsp/v49/0034-8910-rsp-S0034-89102015049005925.pdf
http://www.scielo.br/pdf/rsp/v49/0034-89... ⁾. Considering the importance of the SIPNI for the practice of immunization and the institutionalization of the evaluation of health programs, services and practices, there is a proposal to develop models to evaluate the process of SIPNI implantation. The first stage of this evaluation is treated as accessability assessment (EA), the subject of this article.

EA is developed prior to the evaluation itself and consistently describes a plan for the evaluation, including stakeholder relationships and operational logic, plausibility analysis and the feasibility of its objectives, making it more consistent and credible. The full description of the intervention of the logical and theoretical models of the key issues to be addressed by the evaluation, the evaluation plan and the agreement between the stakeholders is expected⁽⁹9 Thurston W, Ramaliu A. Evaluability assessment of a survivor of torture program: lessons learned. Can J Program Eval [Internet]. 2005 [cited 2017 Mar 15]; 20(2):1-25. Available from: http://www.evaluationcanada.ca/secure/20-2-001.pdf
http://www.evaluationcanada.ca/secure/20... ^-1010 Trevisan MS, Walser TM. Evaluability assessment: improving evaluation quality and use. CA: Sag. Thousand Oaks; 2014.⁾.

Recently, this type of evaluative approach has aroused the interest of many scholars and is increasingly frequent in Brazil, since it allows an in-depth understanding of the most important aspects of the research object and a prior appreciation of the possibilities of further evaluation, maximizing its potential⁽¹¹11 Padilha MA, Oliveira CM, Figueiró AC. Estudo de avaliabilidade do programa academia carioca da saúde: desafios para a promoção da saúde. Saúde Debate [Internet]. 2015 [cited 2017 Feb 26]; 39(105):375-86. Available from: http://www.scielo.br/pdf/sdeb/v39n105/0103-1104-sdeb-39-105-00375.pdf
http://www.scielo.br/pdf/sdeb/v39n105/01...

12 Pereira CN, Luiza VL, Cruz MM. Serviços farmacêuticos na atenção primária no município do Rio de Janeiro: um estudo de avaliabilidade. Saúde Debate [Internet]. 2015 [cited 2017 May 26]; 39(105):451-68. Available from: http://www.redalyc.org/pdf/4063/406341748014.pdf
http://www.redalyc.org/pdf/4063/40634174... ^-1313 Coelho AA, Martiniano CS, Brito EWG, Negrão OGC, Arcêncio RA, Uchôa SAC. Tuberculosis care: an evaluability study. Rev Latino-Am Enferm [Internet]. 2014 [cited 2017 Jun 27]; 22(5):792-800. Available from: http://www.scielo.br/pdf/rlae/v22n5/0104-1169-rlae-22-05-00792.pdf
http://www.scielo.br/pdf/rlae/v22n5/0104... ⁾. A program, in this case the SIPNI, deserves extensive evaluation when it is considered that it operates as expected, is relatively stable over time and there are indications that it is achieving some positive result.

The NIP has as main actors at the local level of care, the nursing team. Thus, the nurse who takes advantage of this methodology can evaluate the actions of services in the vaccination room, identifying fragilities, improvements and possible actions, subsidizing the planning and decision making regarding vaccination activities in the daily services.

OBJECTIVE

To perform the EA of SIPNI, considering the description of the intervention (SIPNI implementation), the elaboration of the theoretical-logical model and the identification of the evaluative questions.

METHOD

Ethical aspects

This study was approved by the Human Research Ethics Committee of the Federal Universidade de São João Del-Rei (CEP/UFSJ).

Type of study and theoretical and methodological reference

An EA, with a qualitative approach, based on the seven elements system proposed by Thurston and Ramaliu⁽⁹9 Thurston W, Ramaliu A. Evaluability assessment of a survivor of torture program: lessons learned. Can J Program Eval [Internet]. 2005 [cited 2017 Mar 15]; 20(2):1-25. Available from: http://www.evaluationcanada.ca/secure/20-2-001.pdf
http://www.evaluationcanada.ca/secure/20... ⁾, are: (a) a description of the program identifying the goals, objectives and activities that constitute it; (b) identification and revision of the documents available in the program; (c) the modeling of available resources, expected activities, expected impacts, and presumed causal connections (Logical Model of the Program); (d) supervising the program, or obtaining a preliminary understanding of how it operates; (e) the development of a Theoretical Model of Evaluation; (f) identification of evaluation users and other key stakeholders; and (g) agreement on the evaluation procedure.

In order to ensure the quality of the research, the evaluation parameters of the Joint Committee on Standards for Educational Evaluation⁽¹⁴14 Joint Committee on Standards for Educational Evaluation. JCSEE. Program Evaluation Standards Statements [Internet]. 2017 [cited 2017 Jul 4]. Available from: http://www.jcsee.org/program-evaluation-standards-statements
http://www.jcsee.org/program-evaluation-... ⁾ (JCSEE) were used: usefulness, feasibility/viability, ownership and precision/accuracy.

Methodological procedures

In the period between October 2016 and January 2017, documents including manuals, regulations, technical standards, reports and materials on SIPNI were analyzed, as well as articles available in PubMed, SciELO, Lilacs and Medline databases. At the same time, visits were made to the Department of Public Health (DPH) of Minas Gerais State (MG) to the Regional Health Superintendence (RHS) of the Western Region and to the Municipal Health Secretariat (MHS) of two MG municipalities that implemented the SIPNI and their respective vaccination rooms, to apprehend the reality of its operationalization and identification of stakeholders (Stakeholders) that could contribute and support the evaluation. Four meetings were held with the participation of two researchers with expertise in immunization and information systems and 10 stakeholders (technical references in immunization and SIPNI in MG. The intentional choice of the State MG and of the municipalities for an on-site visit is justified by the proposal of an evaluation study of the type of implantation analysis⁽¹⁵15 Champagne F, Contandriopoulos AP, Brousselle A, Hartz Z, Denis JL. A avaliação no campo da saúde: conceitos e métodos. In: Brousselle A, Champagne F, Contandriopoulos AP, Hartz Z (Orgs.). Avaliação em saúde: conceitos e métodos. Rio de Janeiro: Fiocruz; 2011.⁾, to be developed by the authors of this article later.

The set of information made possible the elaboration of the theoretical and logical models from the perspective of the authors Champagne et al⁽¹⁵15 Champagne F, Contandriopoulos AP, Brousselle A, Hartz Z, Denis JL. A avaliação no campo da saúde: conceitos e métodos. In: Brousselle A, Champagne F, Contandriopoulos AP, Hartz Z (Orgs.). Avaliação em saúde: conceitos e métodos. Rio de Janeiro: Fiocruz; 2011.⁾ and the agreement of the EA procedure. From the SIPNI’s logical model, the evaluative questions were identified and later, these were validated according to the criteria of relevance, objectivity and clarity. The Delphi technique⁽¹⁶16 Coluci MZO, Alexandre NMC, Milani D. Construção de instrumentos de medida na área da saúde. Ciênc Saúde Colet [Internet]. 2015 [cited 2017 May 26]; 20(3):925-36. Available from: http://www.scielo.br/pdf/csc/v20n3/1413-8123-csc-20-03-00925.pdf
http://www.scielo.br/pdf/csc/v20n3/1413-... ⁾ was used to validate the content of the questions.

29 judges were invited to participate in the panel, among them: 13 PhDs and masters in the academic area and 16 specialists working in HD, DPH, RHS and MHS. Using the LimeSurvey software, each of the judges was asked to evaluate each question, classifying them in an opinion scale: (1) Non-Representative, (2) Representative, but needs revision and (3) Representative item⁽¹⁷17 Lucian R, Dornellas JS. Mensuração de Atitude: proposição de um protocolo de elaboração de escalas. RAC [Internet]. 2015 [cited 2017 Jun 3]; 19(2):157-77. Available from: http://www.scielo.br/pdf/rac/v19nspe2/1982-7849-rac-19-spe2-0157.pdf
http://www.scielo.br/pdf/rac/v19nspe2/19... ⁾. The degree of agreement of the answers was found from the percentage of questions classified in item 3 (representative). Values above 70% were considered as a cutoff point for obtaining the consensus⁽¹⁸18 Pereira RDM, Alvim NAT. Técnica Delphi no diálogo com enfermeiros sobre a acupuntura como proposta de intervenção de enfermagem. Esc Anna Nery Rev Enferm [Internet]. 2015 [cited 2017 Jun 5]; 19(1):174-80. Available from: http://www.scielo.br/pdf/ean/v19n1/1414-8145-ean-19-01-0174.pdf
http://www.scielo.br/pdf/ean/v19n1/1414-... ⁾.

RESULTS

Understanding the intervention: identification and review of available documents, description of the National Immunization Program Information System and obtaining a preliminary understanding of how it operates (Elements a, b, d)

In Brazil, the implantation of the Immunization Information System (IIS) began in 1994. The Brazil›s Information System of the Immunization Program Assessment (SI-API) was the first to be implemented to organize the immunization actions in the national territory, reaching a coverage of 100% in 1997⁽¹⁹19 Brasil. Ministério da Saúde . Fundação Nacional de Saúde e Centro Nacional de Publicação . Manual PNI - Programa Nacional de Imunizações. Brasília: Brasília: Ministério da Saúde; 1998.⁾.

Over the course of 12 years, the SI-API was a means for the dissemination of vaccination data, but it became an insufficient technology in response to the new demands of the NIP. This led to the development of a new IIS in 2010, which brought together, in a single database, the existing subsystems of immunization actions⁽⁵5 Sato APS. National immunization program: computerized system as a tool for new challenges. Rev Saúde Pública [Internet]. 2015 [cited 2016 Dec 10];49:39. Available from: http://www.scielo.br/pdf/rsp/v49/0034-8910-rsp-S0034-89102015049005925.pdf
http://www.scielo.br/pdf/rsp/v49/0034-89... ^,2020 Brasil. Ministério da Saúde. Secretaria de Vigilância a Saúde. Departamento de Informática do SUS. Coordenação Geral do Programa Nacional de Imunizações. Manual do Sistema de Informação do Programa Nacional de Imunizações - SIIPNI. Brasília: Ministério da Saúde; 2014.⁾.

The SIPNI was developed by PNI in partnership with the Department of Information Technology of SUS (DATASUS), with the purpose of aggregating existing systems in a single software and in an interconnected way. It replaced the seven modules or subsystems: Immunobiological Inventory and Distribution Information System (SI-EDI); Information System for the Assessment of Immunobiologicals Used (SI-AEFV); Information System of adverse events following vaccination (SI-AEFV); Information System of the Instrument Evaluation Program (SI-PAIS); Information System of the Evaluation Program of the Supervision Instrument in Vaccine Room (SI-PAISSV); Information System of the Reference Center for Special Immunobiological (SI-CRIE); and the Information System of the Immunization Program Assessment (SI-API)⁽⁶6 Brasil. Ministério da Saúde. Datasus. SI - PNI - Sistema de Informações do Programa Nacional de Imunizações [Internet]. 2017 [cited 2017 Jan 28]. Available from: http://datasus.saude.gov.br/sistemas-e-aplicativos/epidemiologicos/si-pni
http://datasus.saude.gov.br/sistemas-e-a... ⁾.

There are two versions of SIPNI, the desktop (offline mode) and the web (online mode). In 2010, the implementation of the desktop version was begun in Brazilian municipalities. Currently, the web version is still being implemented in some Brazilian cities. Both versions have their advantages and disadvantages. The main difficulty of SIPNI web is the need for good quality internet access and adequate and timely registration of vaccination when the system is off-line. In the desktop version, the main difficulty is to avoid duplicate records, since the vaccination rooms of the same municipality do not communicate with each other, so that an individual registered in a health unit may be re-registered in another unit of the same municipality⁽⁴4 Luhm KR, Waldman EA. Sistemas informatizados de registro de imunização: uma revisão com enfoque na saúde infantil. Epidemiol Serv Saúde [Internet]. 2009 [cited 2017 Jan 14]; 18(1):65-78. Available from: http://scielo.iec.pa.gov.br/pdf/ess/v18n1/v18n1a07.pdf
http://scielo.iec.pa.gov.br/pdf/ess/v18n... ^-55 Sato APS. National immunization program: computerized system as a tool for new challenges. Rev Saúde Pública [Internet]. 2015 [cited 2016 Dec 10];49:39. Available from: http://www.scielo.br/pdf/rsp/v49/0034-8910-rsp-S0034-89102015049005925.pdf
http://www.scielo.br/pdf/rsp/v49/0034-89... ⁾.

The SIPNI is based on the General Coordination of the NIP (GCNIP) that recognizes this technological innovation beyond its potential, an effective means for the dissemination of vaccination data. It is a developed SIS, in order to generate individualized information of quality in the local instance to subsidize the decisions and actions in the scope of its management. The objectives of this study are the identification of vaccinates and their origin, the provision of data on vaccination coverage and the proportion of abandonment and GVAP, as well as the immunobiological assessment used in the network and the Reference Center for Special Immunobiological (CRIE) (BRASIL, 2014 ). Information produced by SIPNI may also be used for academic purposes⁽⁵5 Sato APS. National immunization program: computerized system as a tool for new challenges. Rev Saúde Pública [Internet]. 2015 [cited 2016 Dec 10];49:39. Available from: http://www.scielo.br/pdf/rsp/v49/0034-8910-rsp-S0034-89102015049005925.pdf
http://www.scielo.br/pdf/rsp/v49/0034-89... ⁾.

The expected goals, with the decentralization of SIPNI, include: identification of pockets with low vaccination coverage, improvement of epidemiological surveillance of immunoprevalent diseases and active surveillance of AEFV, as well as favoring studies to evaluate the impact of vaccines in conjunction with other vaccines data base⁽⁵5 Sato APS. National immunization program: computerized system as a tool for new challenges. Rev Saúde Pública [Internet]. 2015 [cited 2016 Dec 10];49:39. Available from: http://www.scielo.br/pdf/rsp/v49/0034-8910-rsp-S0034-89102015049005925.pdf
http://www.scielo.br/pdf/rsp/v49/0034-89... ⁾.

Despite the benefits of the SIPNI, there was little adherence of the municipalities in the first years of implantation. In order to stimulate the implementation of this System, in 2012, a financial incentive was passed to the states, federal district and municipalities, by means of the signing of the agreement signed between states and municipalities⁽²¹21 Brasil. Ministério da Saúde. Repasse financeiro do Ministério da Saúde aos Fundos de Saúde dos Estados. Portaria Nº 2.363, de 18 de outubro de 2012 [Internet]. 2012 [cited 2017 Jun 10]. Available from: http://cosemsrs.org.br/imagens/portarias/por_o2i7.pdf
http://cosemsrs.org.br/imagens/portarias... ^-2222 Brasil. Ministério da Saúde. Repasse financeiro do ministério da saúde aos fundos de saúde dos estados. Portaria Nº 1.779, de 26 de agosto de 2013 [Internet]. 2013 [cited 2017 Jun 10]. Available from: http://bvsms.saude.gov.br/bvs/saudelegis/gm/2013/prt1779_26_08_2013.html
http://bvsms.saude.gov.br/bvs/saudelegis... ⁾. Training recommendations for SIPNI technical references and primary health care (PHC) professionals are added to this. At the time, video-lessons were made available through the channel of the Ministry of Health (MS) and Youtube in order to help and train professionals to use the SIPNI. This conduct was in line with the determination that all municipalities implement the SIPNI by the deadline of December 31, 2013⁽²⁰20 Brasil. Ministério da Saúde. Secretaria de Vigilância a Saúde. Departamento de Informática do SUS. Coordenação Geral do Programa Nacional de Imunizações. Manual do Sistema de Informação do Programa Nacional de Imunizações - SIIPNI. Brasília: Ministério da Saúde; 2014.^,2323 Brasil. Ministério da Saúde. Secretaria de Vigilância em Saúde. Dispõe sobre o prazo final de implantação do SIPNI aos municípios e estados. Ofício Circular 123/2013 [Internet]. 2013 [cited 2017 Jun 10]. Available from: http://www.saude.campinas.sp.gov.br/vigilancia/vacinacao/2016/manual_SIPNI_fev_2014.pdf
http://www.saude.campinas.sp.gov.br/vigi... ⁾.

The timeline presented in Figure 1 shows the main events and the processes that characterize it, expressed in the documents analyzed. In these guiding elements proposed by Thurston and Ramaliu⁽⁹9 Thurston W, Ramaliu A. Evaluability assessment of a survivor of torture program: lessons learned. Can J Program Eval [Internet]. 2005 [cited 2017 Mar 15]; 20(2):1-25. Available from: http://www.evaluationcanada.ca/secure/20-2-001.pdf
http://www.evaluationcanada.ca/secure/20... ⁾, the standard of precision quality is met, since it guaranteed the findings considered correct on the SIPNI.

Identification of stakeholders in the evaluation and agreement on the evaluation procedure (Elements f, g)

This study was proposed and conducted by researchers from the Federal University of São João Del Rei (UFSJ) and had ten adherents engaged in different levels of performance in the SIPNI, being these technical references, nurses and researchers. Interested parties approved and encouraged the evaluation, provided information, documents and electronic files, as well as participated in the work meetings. This involvement made it possible to bring researchers closer to the situation and to the various aspects related to SIPNI. These stages ensure compliance with quality standards: utility (evaluation becomes useful and relevant), ownership (ethics, respect for the right and interest of stakeholders in the evaluation).

In addition, the evaluators acknowledged the involvement of other stakeholders in the evaluation of the SIPNI, and even without their participation in this study, the involvement, interest and participation of each one in the evaluation process was presented (Chart 1).

Development of models for the evaluation of the National Immunization Program Information System (Elements c, e)

After the description of the SIPNI, and obtaining an understanding of its operationalization, the theoretical and logical models of the SIPNI were developed with a focus on the evaluation objective. In the theoretical model, the general context of the NIP as the coordinator of the SIPNI is emphasized, in the external context, which emphasizes the improvement of the immunization actions in the localities; the DATASUS that developed SII as an innovative technology for the transmission of vaccination data; the public and private networks, managers and users of SIPNI; and the Teaching and Research Institutions, users of the information produced and transmitters of knowledge for its improvement. The internal context is the organization of the SIPNI itself, whose purpose is to generate timely and adequate individualized information of the vaccinated (Figure 2).

In order to understand the organization of the SIPNI, the logical model was developed, which consists of an objective image of how the system is supposed to work. It synthesizes the main components of the intervention, from its planning to the expected results⁽²⁴24 Champagne F, Brousselle A, Hartz Z, Contandriopoulos AP. Modelizar as Intervenções. In: Brousselle A, Champagne F, Contandriopoulos AP, Hartz ZMA (Orgs.). Avaliação em saúde: conceitos e métodos. Rio de Janeiro: Fiocruz; 2011.⁾.

The logical model of the SIPNI defines three components to be evaluated, they are: Management of the SIPNI that is subdivided into two subcomponents (Analysis and dissemination of information and Operationalization); Vaccine Registration and Immunobiological Movement. For each of the components and subcomponents described in the logical model, the necessary inputs (resources employed and their organization), the activities (services or goods produced) and the expected results in the short, medium and long term (impact) were defined⁽¹⁵15 Champagne F, Contandriopoulos AP, Brousselle A, Hartz Z, Denis JL. A avaliação no campo da saúde: conceitos e métodos. In: Brousselle A, Champagne F, Contandriopoulos AP, Hartz Z (Orgs.). Avaliação em saúde: conceitos e métodos. Rio de Janeiro: Fiocruz; 2011.⁾ as shown in Figure 3.

This stage ensured the identification of components and presumed causal relationships, as well as the formulation of evaluative questions. The validation of the content of the questions identified (59) showed agreement of 80.6% in relevance, 72.3% in objectivity and 71.5% in clarity of the questions. Of the 59 questions analyzed, two were discarded (degree of agreement less than 70%). In general, the judges reported that the issues were clear, easy to understand, and addressed the activities, resources, and expected outcomes of the SIPNI organization.

The classified questions (57) were arranged in a questionnaire, grouped according to the SIPNI components and the dimensions of structure, process and results⁽¹⁵15 Champagne F, Contandriopoulos AP, Brousselle A, Hartz Z, Denis JL. A avaliação no campo da saúde: conceitos e métodos. In: Brousselle A, Champagne F, Contandriopoulos AP, Hartz Z (Orgs.). Avaliação em saúde: conceitos e métodos. Rio de Janeiro: Fiocruz; 2011.⁾. In these stages of EA, four quality standards were met: precision; the property; utility; and feasibility, since the evaluation takes into account the interests of different groups and the good cost-benefit ratio.

DISCUSSION

The EA was useful for a better understanding of the SIPNI, allowed to clarify its purposes and objectives, the development of the System theory, the elaboration of a model capable of verifying the plausibility of the relations between the components, inputs, activities and results. Theoretical basis of the purposes, means and conditioning factors of the intervention are relevant for the identification of critical nodes relevant to subsequent evaluations⁽²⁵25 Vieira-da-Silva LM. Avaliação de políticas e programas de saúde. Rio de Janeiro: Editora Fiocruz; 2014. 110p.⁾.

Evaluative research has been used to aid the judgment of those who make decisions, since they contribute to the understanding of reality in the daily work, transforming ideas, practices and values. This process of managed management “evidence informed management” transforms and innovates individual and collective learning⁽²⁶26 Denis JL. Institucionalização da avaliação na administração pública. Rev Bras Saúde Mater Infant [Internet]. 2010 [cited 2017 Jun 25];10(Suppl-1):s229-s333. Available from: http://www.scielo.br/pdf/rbsmi/v10s1/20.pdf
http://www.scielo.br/pdf/rbsmi/v10s1/20.... ⁾.

Among the various evaluation strategies⁽¹⁵15 Champagne F, Contandriopoulos AP, Brousselle A, Hartz Z, Denis JL. A avaliação no campo da saúde: conceitos e métodos. In: Brousselle A, Champagne F, Contandriopoulos AP, Hartz Z (Orgs.). Avaliação em saúde: conceitos e métodos. Rio de Janeiro: Fiocruz; 2011.⁾, the EA bases the development of the evaluation plan. This methodological approach allows subsequent evaluations to be developed more easily and reliably, which maximizes their potential and favors the use of resources under evaluation and the knowledge of those involved in the intervention⁽⁹9 Thurston W, Ramaliu A. Evaluability assessment of a survivor of torture program: lessons learned. Can J Program Eval [Internet]. 2005 [cited 2017 Mar 15]; 20(2):1-25. Available from: http://www.evaluationcanada.ca/secure/20-2-001.pdf
http://www.evaluationcanada.ca/secure/20... ⁾.

It should be noted that the evaluation stages carried out in this study favored compliance with the quality standards of the evaluation⁽¹⁴14 Joint Committee on Standards for Educational Evaluation. JCSEE. Program Evaluation Standards Statements [Internet]. 2017 [cited 2017 Jul 4]. Available from: http://www.jcsee.org/program-evaluation-standards-statements
http://www.jcsee.org/program-evaluation-... ⁾ and the objectives of the EE, which include: to agree on the objectives, targets and target population; verify the existence of available data at a reasonable cost and ascertain whether the knowledge generated will be used by the evaluators and other stakeholders in the intervention⁽⁹9 Thurston W, Ramaliu A. Evaluability assessment of a survivor of torture program: lessons learned. Can J Program Eval [Internet]. 2005 [cited 2017 Mar 15]; 20(2):1-25. Available from: http://www.evaluationcanada.ca/secure/20-2-001.pdf
http://www.evaluationcanada.ca/secure/20... ⁾. The EA evaluation stages are tools that enable the researcher to verify the usefulness, feasibility and use of evaluation resources to analyze health programs or policies⁽¹¹11 Padilha MA, Oliveira CM, Figueiró AC. Estudo de avaliabilidade do programa academia carioca da saúde: desafios para a promoção da saúde. Saúde Debate [Internet]. 2015 [cited 2017 Feb 26]; 39(105):375-86. Available from: http://www.scielo.br/pdf/sdeb/v39n105/0103-1104-sdeb-39-105-00375.pdf
http://www.scielo.br/pdf/sdeb/v39n105/01... ^-1212 Pereira CN, Luiza VL, Cruz MM. Serviços farmacêuticos na atenção primária no município do Rio de Janeiro: um estudo de avaliabilidade. Saúde Debate [Internet]. 2015 [cited 2017 May 26]; 39(105):451-68. Available from: http://www.redalyc.org/pdf/4063/406341748014.pdf
http://www.redalyc.org/pdf/4063/40634174... ⁾.

In the course of the research, the external evaluators valued the identification of the stakeholders in the SIPNI evaluation, in view of the possibility of understanding the practices and identifying problems in the daily life. The involvement of people interested in evaluation almost always elucidates problems associated with management, and are usually decision-makers or implementers, who may incorporate the results of the evaluation to improve, extend or modify the intervention⁽²⁷27 Tanaka OY, Tamaki EM. O papel da avaliação para a tomada de decisão na gestão de serviços de saúde. Ciênc Saúde Colet [Internet]. 2012 [cited 2017 Jun 20]; 17(4):821-8. Available from: http://www.scielo.br/pdf/csc/v17n4/v17n4a02.pdf
http://www.scielo.br/pdf/csc/v17n4/v17n4... ⁾. This articulation between internal and external perspectives in an evaluation constitutes the most desirable approach to identify problems and explanations of the evaluated object⁽²⁶26 Denis JL. Institucionalização da avaliação na administração pública. Rev Bras Saúde Mater Infant [Internet]. 2010 [cited 2017 Jun 25];10(Suppl-1):s229-s333. Available from: http://www.scielo.br/pdf/rbsmi/v10s1/20.pdf
http://www.scielo.br/pdf/rbsmi/v10s1/20.... ⁾.

It is also added to the structure of this study, the analysis of documents that presented information consistent and consistent with the purpose of the SIPNI. However, only the manuals of the NIP⁽¹⁹19 Brasil. Ministério da Saúde . Fundação Nacional de Saúde e Centro Nacional de Publicação . Manual PNI - Programa Nacional de Imunizações. Brasília: Brasília: Ministério da Saúde; 1998.⁾ and the SIPNI⁽²⁰20 Brasil. Ministério da Saúde. Secretaria de Vigilância a Saúde. Departamento de Informática do SUS. Coordenação Geral do Programa Nacional de Imunizações. Manual do Sistema de Informação do Programa Nacional de Imunizações - SIIPNI. Brasília: Ministério da Saúde; 2014.⁾ were identified, the other analyzed included the ordinances. There was only one review article on SIPNI⁽⁵5 Sato APS. National immunization program: computerized system as a tool for new challenges. Rev Saúde Pública [Internet]. 2015 [cited 2016 Dec 10];49:39. Available from: http://www.scielo.br/pdf/rsp/v49/0034-8910-rsp-S0034-89102015049005925.pdf
http://www.scielo.br/pdf/rsp/v49/0034-89... ⁾. In addition to the information, on-site visits and meetings were held to expand the description of the SIPNI. Nowadays, the combination of different research techniques in the evaluation is not exclusive or antagonistic, but instead, they translate, in their own way, the articulations between the singular, the individual and the collective⁽²⁶26 Denis JL. Institucionalização da avaliação na administração pública. Rev Bras Saúde Mater Infant [Internet]. 2010 [cited 2017 Jun 25];10(Suppl-1):s229-s333. Available from: http://www.scielo.br/pdf/rbsmi/v10s1/20.pdf
http://www.scielo.br/pdf/rbsmi/v10s1/20.... ^-2727 Tanaka OY, Tamaki EM. O papel da avaliação para a tomada de decisão na gestão de serviços de saúde. Ciênc Saúde Colet [Internet]. 2012 [cited 2017 Jun 20]; 17(4):821-8. Available from: http://www.scielo.br/pdf/csc/v17n4/v17n4a02.pdf
http://www.scielo.br/pdf/csc/v17n4/v17n4... ⁾.

The set of elements already mentioned was essential for the development of the SIPNI logic model. The model was essential to understand the theoretical premises on which the System is based to define exactly what should be measured and what part of its contribution in the observed results⁽²⁴24 Champagne F, Brousselle A, Hartz Z, Contandriopoulos AP. Modelizar as Intervenções. In: Brousselle A, Champagne F, Contandriopoulos AP, Hartz ZMA (Orgs.). Avaliação em saúde: conceitos e métodos. Rio de Janeiro: Fiocruz; 2011.⁾. In addition, the model elaborated in this study provided an increase in internal validity⁽²⁸28 Alves CKA, Freese E, Cesse EAP, Natal S, Bezerra LCA, Felisberto E. Análise da implantação de um programa com vistas à institucionalização da avaliação em uma Secretaria Estadual de Saúde. Rev Bras Saúde Mater Infant [Internet]. 2010 [cited 2017 Jun 20];10(Suppl-1):s145-s156. Available from: http://www.scielo.br/pdf/rbsmi/v10s1/13.pdf
http://www.scielo.br/pdf/rbsmi/v10s1/13.... ⁾ and subsidized the identification of evaluative questions, fundamental in conducting the evaluation.

The clarity of evaluative questions is essential in an evaluative process⁽²⁷27 Tanaka OY, Tamaki EM. O papel da avaliação para a tomada de decisão na gestão de serviços de saúde. Ciênc Saúde Colet [Internet]. 2012 [cited 2017 Jun 20]; 17(4):821-8. Available from: http://www.scielo.br/pdf/csc/v17n4/v17n4a02.pdf
http://www.scielo.br/pdf/csc/v17n4/v17n4... ⁾. The absence of such clarity will result in the accumulation of data and information that may not be useful to those interested in the evaluation. In this sense, the validation of the issues identified through the Delphi technique in this study presented an important strategy to guide pertinent adjustments regarding the content and form of presentation of some questions, which will certainly increase the analytical power of the tool.

With the development of EA, some notes were made on the critical and potential events of the SIPNI, considering the period of observation of the experiences of its implantation until 2017. The SIPNI has potential that can improve the care given to the vaccinated person, immunization actions offered by the team that works in the vaccination room and contribute to the reduction of the rate of illness due to immunoprevalent diseases⁽⁵5 Sato APS. National immunization program: computerized system as a tool for new challenges. Rev Saúde Pública [Internet]. 2015 [cited 2016 Dec 10];49:39. Available from: http://www.scielo.br/pdf/rsp/v49/0034-8910-rsp-S0034-89102015049005925.pdf
http://www.scielo.br/pdf/rsp/v49/0034-89... ^,2929 Novick LF. Immunization information systems. J Public Health Manag Pract [Internet]. 2014 [cited 2017 Jun 20];00(00):1-2. Available from: https://www.thecommunityguide.org/vaccines/vpd-jphpm-ed-IIS.pdf
https://www.thecommunityguide.org/vaccin... ⁾.

One of these potentials is the possibility of monitoring the vaccination status of the people, which allows deciding on the administration of vaccines and active search of the individuals with delayed vaccine situation. Another important point is the qualified management of public and private resources when using the System, since this technology provides a reduction of administrative expenses through stock control and immunobiological losses. In addition, it supports the vaccinator to decide on administration of vaccines, which avoids wasteful doses of immunobiological administered unnecessarily. With this, it is able to assist in the operationalization of immunization actions⁽⁵5 Sato APS. National immunization program: computerized system as a tool for new challenges. Rev Saúde Pública [Internet]. 2015 [cited 2016 Dec 10];49:39. Available from: http://www.scielo.br/pdf/rsp/v49/0034-8910-rsp-S0034-89102015049005925.pdf
http://www.scielo.br/pdf/rsp/v49/0034-89... ^,2929 Novick LF. Immunization information systems. J Public Health Manag Pract [Internet]. 2014 [cited 2017 Jun 20];00(00):1-2. Available from: https://www.thecommunityguide.org/vaccines/vpd-jphpm-ed-IIS.pdf
https://www.thecommunityguide.org/vaccin... ^-3030 Stockwell MS, Fiks AG. Utilizing health information technology to improve vaccine communication and coverage. Hum Vaccin Immunother [Internet]. 2013 [cited 2017 Jun 21]; 9(8):1802-11. Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3906285/
https://www.ncbi.nlm.nih.gov/pmc/article... ⁾.

The information produced enables the identification of pockets with low coverage, which provides the population with equal access to immunobiological and, consequently, reduction of the rate of illness due to immunoprevalent diseases. They support the planning of immunization actions, subsidizing the calculation and monitoring of vaccination coverage, abandonment rates, as well as providing subsidies for the surveillance of the AEFV. In addition, the systematization of vaccination records and the completeness of data positively favor the use of this information in scientific research, which will enhance a description and evaluation that is closer to reality⁽⁴4 Luhm KR, Waldman EA. Sistemas informatizados de registro de imunização: uma revisão com enfoque na saúde infantil. Epidemiol Serv Saúde [Internet]. 2009 [cited 2017 Jan 14]; 18(1):65-78. Available from: http://scielo.iec.pa.gov.br/pdf/ess/v18n1/v18n1a07.pdf
http://scielo.iec.pa.gov.br/pdf/ess/v18n... ^-55 Sato APS. National immunization program: computerized system as a tool for new challenges. Rev Saúde Pública [Internet]. 2015 [cited 2016 Dec 10];49:39. Available from: http://www.scielo.br/pdf/rsp/v49/0034-8910-rsp-S0034-89102015049005925.pdf
http://www.scielo.br/pdf/rsp/v49/0034-89... ^,2929 Novick LF. Immunization information systems. J Public Health Manag Pract [Internet]. 2014 [cited 2017 Jun 20];00(00):1-2. Available from: https://www.thecommunityguide.org/vaccines/vpd-jphpm-ed-IIS.pdf
https://www.thecommunityguide.org/vaccin... ^,3131 Domingues CMAS, Teixeira AMS. Coberturas vacinais e doenças imunopreveníveis no Brasil no período 1982- 2012: avanços e desafios do Programa Nacional de Imunizações. Epidemiol Serv Saúde [Internet]. 2013 [cited 2017 Jul 21]; 22(1):9-27. Available from: http://scielo.iec.pa.gov.br/pdf/ess/v22n1/v22n1a02.pdf
http://scielo.iec.pa.gov.br/pdf/ess/v22n... ⁾.

However, this information is underutilized in planning the actions and management of these services. Most of the time, the planning takes place according to the spontaneous demand of the users. Another important fact observed is that managers have little knowledge about IIS and its purposes. The view of these managers is based on the idea that computerized systems are only compulsory, bureaucratic instruments whose function is to capture data from health units and their areas of coverage and refer them to the municipal, state and federal levels⁽³²32 Lima KWS, Antunes JLF, Silva ZP. Percepção dos gestores sobre o uso de indicadores nos serviços de saúde. Saúde Soc [Internet]. 2015 [cited 2017 Feb 18]; 24(1):61-71 Available from: http://www.scielosp.org/pdf/sausoc/v24n1/0104-1290-sausoc-24-1-0061.pdf^-3333 Cavalcante RB, Bernardes MFVG, Gontijo TL, Guimarães EAA, Oliveira VC. Sistema de informação da atenção básica: potencialidades e subutilização no processo decisório. Cogitare Enferm [Internet]. 2013 [cited 2017 May 26]; 18(3):460-7. Available from: http://revistas.ufpr.br/cogitare/article/view/33555/21054
http://revistas.ufpr.br/cogitare/article... ⁾.

Incompleteness and consistency of data, fragmentation in health care, financing of software and financing are the main obstacles to the use of these systems⁽³⁴34 Montague E. The promises and challenges of health information technology in primary health care. Prim Health Care Res Dev [Internet]. 2014 [cited 2017 Jul 21]; 15(3):227-30. Available from: https://www.cambridge.org/core/services/aop-cambridge-core/content/view/A041F6E9A40597F5FFBA9DAB717225AA/S1463423614000231a.pdf/promises_and_challenges_of_health_information_technology_in_primary_health_care.pdf
https://www.cambridge.org/core/services/... ^-3535 Sheikh A, Jha A, Cresswell K, Greaves F, Bates DW. Adoption of electronic health records in UK hospitals: lessons from the USA. Lancet [Internet]. 2014 [cited 2017 Jul 21]; 384(9937):8-9. Available from: http://www.thelancet.com/journals/lancet/article/PIIS0140-6736(14)61099-0/fulltext
http://www.thelancet.com/journals/lancet... ⁾. The training of health care professionals and health surveillance is discussed as one of the difficulties for the implantation of SIPNI in Brazil. It is necessary to integrate actions at the municipal, state and national levels of health to carry out training, training, courses and/or workshops for operationalization and use of this SIS as a management tool in the implementation of NIP actions⁽⁵5 Sato APS. National immunization program: computerized system as a tool for new challenges. Rev Saúde Pública [Internet]. 2015 [cited 2016 Dec 10];49:39. Available from: http://www.scielo.br/pdf/rsp/v49/0034-8910-rsp-S0034-89102015049005925.pdf
http://www.scielo.br/pdf/rsp/v49/0034-89... ⁾.

In the EA, it was possible to identify the problematic situation: the new challenges of the SIPNI regarding the activities and responsibilities of the local services resulting from its implementation, incorporating the operationalization of the System, which includes: registration of the vaccinated person, immunobiological movement, use and dissemination of the information produced. It will also identify if the results of the SIPNI will be used and if there is an indication of its sustainability⁽⁹9 Thurston W, Ramaliu A. Evaluability assessment of a survivor of torture program: lessons learned. Can J Program Eval [Internet]. 2005 [cited 2017 Mar 15]; 20(2):1-25. Available from: http://www.evaluationcanada.ca/secure/20-2-001.pdf
http://www.evaluationcanada.ca/secure/20... ⁾. In this regard, for SIPNI to be effective and sustainable, it is necessary to establish the institutional and individual commitment, previously established, to adjust the scope of work, to redefine goals, to consider the financial, administrative and technical aspects, besides the time dimension of the implementation.

Study limitations

The shortage of manuals, regulations and articles on the SIPNI made it difficult to construct the modeling of this technological innovation, which can impact the elaboration of the questions and, consequently, the point of cut of the consensus of the values of the degree of agreement. In the theoretical review, a lack of instruments that assess SIS in all its dimensions (structure, process and results) were observed in Brazil and in other countries. This practice is not yet institutionalized in services, and results evaluations remain as coverage, completeness and reliability of the data⁽²⁹29 Novick LF. Immunization information systems. J Public Health Manag Pract [Internet]. 2014 [cited 2017 Jun 20];00(00):1-2. Available from: https://www.thecommunityguide.org/vaccines/vpd-jphpm-ed-IIS.pdf
https://www.thecommunityguide.org/vaccin...

30 Stockwell MS, Fiks AG. Utilizing health information technology to improve vaccine communication and coverage. Hum Vaccin Immunother [Internet]. 2013 [cited 2017 Jun 21]; 9(8):1802-11. Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3906285/
https://www.ncbi.nlm.nih.gov/pmc/article... ^-3131 Domingues CMAS, Teixeira AMS. Coberturas vacinais e doenças imunopreveníveis no Brasil no período 1982- 2012: avanços e desafios do Programa Nacional de Imunizações. Epidemiol Serv Saúde [Internet]. 2013 [cited 2017 Jul 21]; 22(1):9-27. Available from: http://scielo.iec.pa.gov.br/pdf/ess/v22n1/v22n1a02.pdf
http://scielo.iec.pa.gov.br/pdf/ess/v22n... ⁾.

Contributions to nursing and public policy

The EAs are seen with great relevance by the international scientific community and, therefore, the present study can add to the Brazilian scientific community new theoretical and methodological references in the evaluation of services, programs and health systems. And for nursing, which is exclusively responsible for the activities performed in vaccination rooms in the Brazilian public health service, the effective implementation of the SIPNI will favor the optimization of the immunization work process, reducing the time spent with vaccination records and unnecessary doses administration; and the vaccine room safety with post-vaccination adverse event monitoring and the updated record of the vaccinated.

FINAL CONSIDERATIONS

The EA favored the planning of the SIPNI evaluation, helping to determine the purposes and the evaluative focus of the system. Thus, EA is considered an appropriate evaluation strategy for the planning of subsequent evaluations of the SIPNI. Through this preparatory phase, it was possible to justify, scientifically, the decision to evaluate its implementation and also to know its organization and its operation. The logical model and the matrix of measures will be fundamental for the development of evaluative researches to know to what extent the objectives of the SIPNI are being achieved and what the advances and the challenges observed in the implantation process.

REFERENCES

Publication Dates

History