1308_EN.html

A documentary study on the evaluation of two result measurement indicators in corporate universities

Hellen Costa Ramos

hellencost@gmail.com

State Secretary of Education – SEEDUC, Rio de Janeiro, Rio de Janeiro, Brazil.

Sergio Murilo de Souza Garcia

sergiogarcia@id.uff.br

Aleph Educacional, Niterói, Rio de Janeiro, Brazil.

Leonardo Soares Francisco de Almeida

leonardosoares@id.uff.br

Salgado Filho University – Universo, Rio de Janeiro, Rio de Janeiro, Brazil.

Louise Rosa Araujo

louiserosaaraujo@gmail.com

State Secretary of Education – SEEDUC, Rio de Janeiro, Rio de Janeiro, Brazil.

Rosana Therezinha Queiroz Oliveira

rosanatherezinha@id.uff.br

Oswaldo Cruz Foundation – Fiocruz, Rio de Janeiro, Rio de Janeiro, Brazil.


ABSTRACT

Highlights: this article aims to identify and analyze the indicators of quality measurement used by the Corporate University of Banco do Brasil.

Objective: to verify that these indicators contribute to leverage the education and knowledge essential to the corporate training system.

Methodology: an exploratory and documentary study developed through a systematic review of the literature to support the research until the year 2017.

Results: through the data referring to the indicators analyzed and measured in the case study, it was noticed that, although they have evolved to improve the way to disseminate the knowledge necessary to leverage competitive advantage, there is still no adequate method to measure the results of corporate universities due to the difficulties in the steps presented by Kirkipatrick and Return on investment (ROI). Limitations of research: for the literature review, the study uses the following resources: material published in periodicals in the Scopus database, books, scientific articles; dissertations; theses; networks, that is, material accessible to the general public and which provides analytical tools.

Practical implications: due to the fact that Banco do Brasil SA is a financial institution constituted by a mixed company, its net sales may be the result of its products and not necessarily the investment made in its employees.

Originality/Value: through a documental investigation, the study points out that there is a gap in the literature on the subject of performance evaluation in corporate universities.

Keywords: Corporative education; Corporate University; Performance indicators.


INTRODUCTION

Organizations in general, both nationally and internationally, face endemic changes in the external environment in multiple dimensions[1], which causes rapid changes in the internal environment.

Organizational management systems need to be able to respond to changes at a rapid pace. The focus on change makes the management of knowledge come to be seen as a decisive strategy to meet the current changes. According to Buogo et al. (2016), knowledge and information are the main assets of an organization, which turns them into a competitive advantage. Given the dependence that organizations have on these assets, they need to be addressed in a systematic and strategic way. Corporate education has emerged, in this scenario, to structure organizational knowledge, leading it to be used as a competitive advantage.

For Schröeder et al. (2011), when pure information alone does not guarantee success, it is necessary for the organization to be able to lapse the information it receives by transforming it into knowledge, so that, through the sharing of this knowledge, it is possible to achieve originality, greater competitive advantage and productive innovation.

Thus, to obtain survival in the corporate market, organizations need to develop a process of "creating" new knowledge continuously: technical knowledge and human insight; explicit knowledge and tacit knowledge.

In this context emerges the corporate university, which should focus on continuous learning in order to anticipate the demands of the highly competitive labor market.

Cappiello and Pedrini (2017) define that corporate universities have a hybrid nature, since they are institutions of higher education that act in the mold of a business unit.

This exposes the question: through which performance indicators are the results measured?

The objective of this study was to perform an analysis of two models of performance evaluation indicators - Kirkipatrick and Return on investment (ROI) - used by corporate universities, comparing the pertinent literature with the Corporate University of Branco do Brasil (UniBB).

LITERATURE REVISION

Historic evolution

The first corporate university came into being in the United States in the 1950s, with the initiative of Jack Welch of General Electric in 1955, when creating the Crotonville Management Development Institute, but its diffusion took place from the end of the 1980s (Ramos, 2008). In Brazil, in turn, they are more recent. Although they emerged in the early 1990s, it was only in the middle of that decade that it was possible to note their adoption by a larger number of companies.

Corporate universities emerge from a perspective of sustaining competitive advantage and stimulating continuous learning and performance of human values and organizations (Meister, 1999), that is, to invest in the improvement of the employees and to guarantee excellence as the main organizational goal the (Borges, 2007).

According to Eboli (2004, p. 186), in Brazil, the tendency of organizations to establish such universities began with an increase in the perception by companies of the "need to renew their traditional training and development centers in order to contribute with effectiveness and success for the business strategy, adding value to the result of the business".

According to Kelly et Oliveira (2014), through the educational apprenticeships offered through corporate universities, it is feasible to expand the skills and abilities necessary for a better performance of the activities.

From training to corporate university

Historically, following the organizational principles of Taylonism, in a world with relative stability greater than the present, the Training and Development Departments are organized through the reproduction of work and knowledge, the only factors considered capable of guaranteeing the survival of the company. In this context, the main objectives of the programs were to develop specific skills, emphasizing individual needs, acting at the tactical level and without interaction with the company's strategic goals (Eboli, 2004).

According to Carvalho et al. (2014), constant changes and new technologies, with new methods and processes, make the training and development sectors partners and act on the sustainability of the organization. Cruz et al. (2017) commented that, for a long time, organizations saw the area of human resources not as a partner, but as passive, although fundamental to the organization.

Sudhakar et Basariya (2017) define training as the process of providing employees with unique skills or helping them to correct gaps in their performance. Development, in turn, is an effort to provide employees with the capabilities that the company will need in the future.

According to Silva (2008), the profile of the target audience of the training and development sectors is characterized as broad, not restricted to the internal public, which does not contemplate the qualification/capacity-building perspective of corporate universities, which have a specific target audience that, even if it is formed by professionals from different sectors/areas, are aligned with the mission and values of the organization.

Meister (1999) presents differences between the old concept of Training Department and those exercised after the creation of corporate universities, as shown in Table 1.

Table 1. Training and Development versus Business University

Figura1

Source: Meister (1999)

As observed, Meister (1999) characterizes the training and development in a corporate university as "active", that is, the proposition of courses arises from the business strategy, anticipating to the demands; "centralized", convergent to specific areas, and; aimed mainly at strategic/intangible knowledge.

Cislaghi et al. (2015) comment that developing the skills and competencies in the individuals of an organization can positively favor the strategic alignment of the organization.

Mzimela et Chikandiwa (2017) state that, with efficient training and development, organizations have a chance to gain competitive advantage in business units.

Corporate University Concepts

Table 2 presents the definition of corporate university by some authors.

Table 2. Concepts of corporate universities

Figura2

Source: Prepared by the authors

It should be noted that the corporate model of organizational learning is not exempt from threats from the external market, and can generate problems due to strategic and operational inadequacies, which lead to poor institutional acceptance by employees of the institutions sponsoring this model.

Silva et Sá (2017), after searching the literature for definitions of current corporate university models, found that there are currently seven guidelines that strengthen and structure corporate education: scope, interconnection, stakeholder recognition, technology, focus, level, and knowledge management. This perspective is in line with the ideas of Meister (1999) and Eboli (2004), who consider corporate universities as a system of people development based on competence management.

Indicators of performance evaluation used by corporate universities

The Kirkpatrick Model

Donald Kirkpatrick (1994) proposed a system to gauge the specific value of training offered by organizations, which has become one of the best known methods of outcome measurement. The Kirkpatrick model proposes four measurement levels for training (Table 3).

Table 3. Adapted from Kirkpatrick (1998)

Figura3

Source: Palmeira, 2006, p. 15

The four levels evaluated, according to the Kirkpatrick method, are considered sequentially; therefore, each level has special importance; however, as these levels are reached, moving to the next, the process becomes more complex and acute, increasing the need for priorities and the time for its development. On the other hand, it provides valuable information every time it is applied (Ramos et al., 2016).

Kovářová et Šimková (2014) consider that the Kirkpatrick Assessment Model is a unique approach to evidence-based learning, strengthening student satisfaction and enabling students to achieve learning outcomes.

For Hites et al. (2014), before applying the model proposed by Kirkpatrick, it is necessary to understand the fundamental principles of each level and its application to evaluate the results of the training. In this way, performance evaluation should translate the alignment between organizational strategy and people's valuation, so that users realize that the training activity adds value to the business.

Deodhar and Powdwal (2017) argue that if the organizational culture is not configured for behavioral transformations, participants may not be able to apply what they have learned in the organization.

Guerci et al. (2010) suggest that much of the training assessment focuses on a single interested party, the stakeholder, and this practice is strongly based on Kirkpatrick's hierarchical model.

Return on investment (ROI)

Jack Phillips, in 1996, added another level to the Kirkpatrick model:

The Jack Phillips model emerged from the Kirkpatrick model rating scale. It was from Kirkpatrick's 4th level that Phillips realized that there was a possibility of adding one more level of evaluation that was complete and improved, and that would add great impact to the business of the organization (Andrade et Silva, 2010, p. 6).

Botchkarev et al. (2011) define ROI as a performance measure that evaluates the effective outcome of an investment, as shown in Figure 1.

Figure 1. Return on investment (ROI) Model

Figura4

Source: Andrade et Silva (2010)

Andrade et Silva (2010) define ROI as the return that the organization's top management expects to receive from the training area, measuring the result of the investment in training and development.

For Paduam et. al. (2015), the ROI construction process is the fundamental point for the organization; however, each company can customize the method adapting it to its reality.

ROI is a mathematical calculation whose result can be obtained by using the equation represented below (Botchkarev et al., 2011):

Figura5

METHODOLOGY

The present research had as methodological aspect the exploratory drawing and the documentary as predominant of its conception. The theoretical references on the methodology used were searched in the works of the following authors: Lakatos et Marconi (1992; 2002).

The documentary research was carried out with the public data available on the UniBB website, accessed in May 2017. The purpose was to analyze the university's evaluation model as a strategy to attract, train and develop people, as well as retention of their talents.

For the literature review, the study used the following eligibility criteria: material published until 2017 in journals indexed in the Scopus database; books; scientific articles; dissertations; theses; and electronic networks, that is, material accessible to the general public and that provides analytical tools for any other type of research (Vergara, 2003).

With regard to documentary research, the theoretical-methodological basis of the study is carried out through research on the historical evolution of corporate universities, differences between classic training and corporate university, corporate university concepts, the evaluation model advocated by Kirkpartik, ROI, and the critical analysis of the theoretical foundation in comparison with the experience of UniBB.

The methodological flow proposed for this research is shown in Figure 2.

Figure 2. Evolution and construction of the research method

Figura6

The experience of UniBB will be presented when trying to identify the performance indicators used by it.

ANALYSIS OF RESULTS

The documentary research conducted led to the consultation on the organization Banco do Brasil S.A., where the corporate education is practiced through UniBB.

In the following items part of the story, the form adopted to evaluate results, that is, the criterion used to measure their effectiveness will be presented.

The data were collected on the organization's website.

Monitoring and management indicators - Results of the UniBB

Although BB's Corporate Education System has been in existence since 1965, its corporate university (UniBB) was created on July 11, 2002, with the purpose of continuing the evolution of this education to help its employees to improve their skills, to grow professionally and to collaborate to carry out the business (Banco do Brasil, 2017a).

The role of UniBB is:

Develop human and professional excellence through the creation of value in educational solutions, contributing to the improvement of organizational performance and the strengthening of the institutional image of Banco do Brasil. To help its employees improve their talents, grow professionally and contribute to the business. Therefore, on a growing scale, Banco do Brasil has defined its strategic directions in line with the principles of socio-environmental responsibility and, in particular, issues of diversity and accessibility as inalienable human rights, besides factors that add up to the list of conditions essential to the optimal functioning of organizational life (Banco do Brasil, 2017a).

In its Web page, it is mentioned that UniBB has already won the following awards:

2nd place in the category Corporate University of the Year - Global at the 2016 Cubic Awards; International Award promoted by IQPC - International Quality and Productivity Center in partnership with CLN - Corporate Learning Networking. The Cubic Awards (Corporate University Best-in-Class) is a world-class award recognizing the best practices and corporate education programs in the world (Banco do Brasil, 2017a).

The awards received, recognizing their merits, are a return of the importance of the fundamental role played by UniBB.

Table 4 shows the number of hours spent in training in the period 2014-2016 at UniBB.

Table 4. Hours spent in training by the Corporate University of the Banco do Brasil in the period 2014-2016

Figura7

Source: Banco do Brasil, 2017b

According to Guimarães (2008), it should be noted that the indicator of hours of training, when analyzed in isolation, reaches only the "effort" spent by the human resources sector, indicating that the latter worked much or little in terms of training. For better analysis it is therefore indispensable to evaluate the results of the training performed by some indicator that effectively measures the results.

According to the data obtained in its 2016 annual report, available on its website (Banco do Brasil, 2017b), UniBB evaluates its overall return results: “adopting global metrics to quantitatively measure the business benefits of its human capital investment programs”, by means of Human Capital Value Added (HCVA) in training and development; ROI in human resources; Human Economic Value Added (HEVA); and net ROI with training and development.

The overall results of UniBB's return are presented in Table 5.

Table 5. Global Results of Return of the Corporate University of Banco do Brasil in 2015 and 2016

Figura8

Source: Banco do Brasil, 2017b

Analyzing Table 5, it can be seen that the result obtained by the net ROI with training and development in the year 2016 was lower than the result presented in the year 2015.

Faber et al. (2002), comment that there are criticisms of the ROI performance evaluation model, since they are biased as to the results.

CONCLUSION

The research, developed and presented in this manuscript, brings the case of Banco do Brasil, based on its history and business development, as an example of contribution of management by competencies for the development of corporate education systems of organizations.

A corporate university is an increasingly necessary organizational space to provide professional training to its employees and, substantially, an instrument to effectively manage knowledge within an organization (Scarso, 2017).

Banco do Brasil, in its trajectory, demonstrates that the management of professional competencies is an indispensable activity and an essential input for the adequate alignment of the training systems with the organizational strategies.

Thus, it can be said that the diagnosis and the evaluation of competences offer the opportunity of convergence between employees and company, favoring the continued growth of human capital and the achievement of institutional objectives.

The methodology used to measure Banco do Brasil's results is centered on the model of net ROI with training and development, whose calculation indicated smaller values in 2016, when compared to those of 2015.

It was also possible to see that the structuring of the Banco do Brasil's human resources training and development process, defining the policies, norms and procedures for its operation, can effectively enable the training of employees and company.

In summary, the research made it possible to discern that one of the biggest challenges facing companies today is to create effective indicators for measuring the results obtained with investments in training.

The indicators used traditionally reflect only the internal reality and help little in the understanding of how much the business really benefited from the training. Because Banco do Brasil S.A. is a financial institution constituted by a mixed company, its net sales may be the result of its products and not necessarily the investment made in its employees.

Thus, the results of initiatives should take into account the evaluation of educational programs, the measurement of internal and external results indicators and the use of technology to support the diffusion of knowledge in social networks.


REFERENCES

Alperstedt, C. (2001), “Universidades Corporativas: discussão e proposta de uma definição”, Revista de Administração Contemporânea, Vol. 5, No. 3, pp. 149-165.

Andrade, R. J. F.; Silva, N. F. (2010), “Roi – uma análise comparativa entre os métodos de avaliação sobre o retorno do investimento em treinamento”, apesentado no VII Congresso Virtual Brasileiro de Administração – CONVIBRA, 19-21 nov. 2010, disponível em: http://www.convibra.org/upload/paper/adm/adm_1357.pdf. (Acesso em 14 de abril de 2017).

Banco do Brasil (2017a), Universidade Corporativa, disponível em: https://www.bb.com.br/portalbb/home1,139,139,23,0,1,8.bb (Acesso em 09 maio 2017).

Banco do Brasil (2017b), Relatório Anual 2016, disponível em: https://www.bb.com.br/docs/pub/siteEsp/ri/pt/dce/dwn/relan2016.pdf (Acesso em 09 maio 2017).

Borges, C. V. A.; Zorzenon, V.; Chiacchio, S. S. R. (2017), “Treinamento e desenvolvimento com foco no clima organizacional, “Revista Espacios, Vol. 38, No. 11, pp. 8, disponível em: http://www.revistaespacios.com/a17v38n11/17381108.html (Acesso em 20 maio 2017).

Borges, M. (2007), “Excelência Corporativa: Empresas descobrem importância de estimular funcionários a estudar”, publicado em 31 dez. 2007.

Botchkarev. P. A.; Andru, P.; Chiong, R. (2011), “A Return on Investment as a Metric for Evaluating Information Systems”, Interdisciplinary Journal of Information Knowledge and Management, Vol. 6, No. 16, pp. 254-269, disponível em: http://www.ijikm.org/Volume6/IJIKMv6p245-269Botchkarev566.pdf (Acesso em 25 maio 2017).

Buogo, M.; Yanzer, A.; Basso Junior, E. (2016), “Methodologies of knowledge management Considering best practice for information security”, Atas da Conferência da Associação Portuguesa de Sistemas de Informação, Vol. 16, pp. 69-83. DOI: 10.18803/capsi.v16.69-83.

Cappiello, G.; Pedrini, G. T. (2017), “The performance evaluation of corporate universities”, Tertiary Education and Management, Vol. 23, No. 3, pp. 304-317. DOI: 10.1080/13583883.2017.1329452.

Carvalho, M B et al. (2014), “Treinamento e desenvolvimento como parcerias da empregabilidade sustentabilidade da organização”, Revista Caderno de Graduação – Ciências Humanas e Sociais, Vol. 2, No. 1, pp. 25-36, disponível em: https://periodicos.set.edu.br/index.php/cadernohumanas/article/view/988/700 (Acesso em 19 maio 2017).

Cislaghi, T. P.; D'arisbo, A.; Ribeiro, J. M. C.; Barcellos, P. F. P. (2015), “Estratégia Empresarial, Competências e Aprendizagem Organizacional: Um Estudo de caso da Reestruturação do Grupo Nova Energia”, Revista de Administração IMED, disponível em: https://seer.imed.edu.br/index.php/raimed/article/view/468/573 (Acesso em 19 maio 2017).

Deodhar, M.; Powdwal, S. (2017), "Impact of continuing education programs (CEPs) on LIS professionals in academic libraries in Mumbai, India", Library Management, Vol. 38, No. 2/3, pp. 117-130, available from: https://doi.org/10.1108/LM-07-2016-0051 (Access 04 nov. 2018).

Eboli, M. (1999), “Gestão do Conhecimento como Vantagem Competitiva”, In: Eboli, M. (org.), Coletânea Universidades Corporativas: educação para as empresas do século XXI, Schmukler, São Paulo.

Eboli, M. (2004), “Educação Corporativa no Brasil: mitos e verdades”, Gente, São Paulo.

Faber, B.; Bekins, L.; Karis, B. (2002), “Using corporate-based methods to assess technical communication programs”, Journal of Technical Writing and Communication, Vol. 32, No. 4, pp. 307-336. DOI: 10.2190/T2HC-KXTD-7YFK-4PFV.

Guerci, M.; Bartezzaghi, E.; Solari, L. (2010), “Training evaluation in Italian corporate universities: A stakeholder-based analysis”, International Journal of Training and Development, Vol. 14, No. 4, pp. 291-308. DOI: 10.1111/j.1468-2419.2010.00359.x

Hites, L. S.; Sass, M. M.; D'Ambrosio, L. et al. (2014), “The Preparedness and Emergency Response Learning Centers: Advancing standardized evaluation of public health preparedness and response trainings“, Journal of Public Health Management and Practice, Vol. 20, No. Suppl. 5, pp. S17-S23, disponível em https://www.ncbi.nlm.nih.gov/pubmed/25072484. (Acesso em 18 maio 2017).

Kelly, G.; Oliveira, A. (2014), “O impacto da universidade corporativa no âmbito organizacional: estudo de caso na Concessionária Chevrolet”, Gestão & Aprendizagem, Vol. 3, No. 1, pp. 64-86. DOI: https://doi.org/10.23179/2317-9082

Kirkpatrick, D. (1994), Evaluating Training Programs: The Four Levels, Berrett-Koehler, San Francisco.

Kovářová, P.; Šimková, G. (2014), “Abordagem de Aprendizagem Baseada em Evidências na Avaliação da Educação em Alfabetização Informacional”, In: Kurbanoğlu S., Špiranec S., Grassian E., Mizrachi D., Catts R. (eds), Information Literacy. Aprendizagem ao Longo da Vida e Cidadania Digital no Século XXI. ECIL 2014. Comunicações em Ciência da Computação e Informação, Vol. 492, Springer.

Lakatos, E. M.; Marconi, M. A. (2002), “Técnicas de Pesquisa: planejamento e execução de pesquisas, amostragem e técnicas de pesquisas, elaboração e interpretação de dados”, 5. ed., Atlas, São Paulo.

Meister, J. (1999), “Educação Corporativa: a gestão do capital intelectual através das universidades corporativas”, Makron, São Paulo.

Mzimela, T.; Chikandiwa, C. T. (2017), “Employee training and development practices in the Tourism and Leisure sector in KwaZulu-Natal, South Africa African”, Journal of Hospitality, Tourism and Leisure, Vol. 6, No. 4, pp. 1-17, disponivel em: http://www.ajhtl.com/uploads/7/1/6/3/7163688/article_22_vol_6__4__2017.pdf (Acesso em 26 maio 2017).

Paduam, T. C.; Fabri, J. A.; L´Erario, A (2015), “Modelo para Calcular o Retorno sobre Investimento após a Implantação de Software Programa de Pós-Graduação em Informática, Universidade Tecnológica Federal do Paraná”, Revista de Sistemas de Informação da FSMA, No. 15, pp. 40-51, disponível em: http://www.fsma.edu.br/si/edicao15/FSMA_SI_2015_1_Estudantil_3.pdf. (Acesso em 22 de maio de 2017).

Palmeira, C. G. (2006), “Avaliação de Resultados: retorno do investimento”, In: Manual de Treinamento e Desenvolvimento – Processos e Operações, Person Prentice Hall, Rio de Janeiro, pp. 1-14.

Ramos, H. C. et al. (2008), “Transição de treinamento e desenvolvimento (T&D) para universidade corporativa”, IV Congresso Nacional de Excelência em Gestão, 31 jul. 2008, disponível em: http://www.inovarse.org/filebrowser/download/8771 (Acesso em: 09 fev. 2017).

Scarso, E. (2017), "Universidades corporativas como ferramentas de gestão do conhecimento", Revista VINE de Sistemas de Informação e Gestão do Conhecimento, Vol. 47, No. 4, pp. 538-554, disponível em: https://doi.org/10.1108/VJIKMS-12-2016-0074 (Acesso em: 09 fev. 2017).

Schröeder, C. S.; Antunes, M. P.; Oliveira, J. L. (2011), “Gestão do Conhecimento Corporativo: Um Fator de Competitividade para as Organizações”, Revista de Administração IMED, Vol. 1, No. 1, pp. 29-53.

Silva, D. R. (2008), “Educação Corporativa”, disponível em: http://www.fecap.br/PortalNovo/Index.asp (Acesso em 20 maio 2017).

Silva, J. R. S.; Almeida, C. D.; Guindani, J. F. (2009). “Pesquisa documental: pistas teóricas e metodológicas”, Revista Brasileira de História & Ciências Sociais, Ano I, No. I, available from: https://www.rbhcs.com/rbhcs/article/viewFile/6/pdf (Acesso em 10 maio 2017).

Silva, T. C.; Sá, F. P. (2017), “Corporate University: Guidelines that must be managed for the implementation of the model“, Espacios, Vol. 38, No. 10, pp. 13, disponível em: http://www.revistaespacios.com/a17v38n10/17381015.html (Acesso em 27 maio 2017).

Sudhakar, R.; Basariya, S. R. (2017), “Training and development programmes in IT company and its role in aggrandizing employee performance”, International Journal of Civil Engineering and Technology, Vol. 8, No. 8, pp. 1747-1753, disponível em http://paper.researchbib.com/view/paper/158170 (Acesso em 29 maio 2017).

Vergara, S. C. (2003), Projetos e Relatórios de Pesquisa em Administração, Atlas, São Paulo.

Vianna, M. A. F. (2003), “Universidade Corporativa: uma visão de hoje”, Revista Aprender Virtual, São Paulo, abril 2003, disponível em: http://www.institutomvc.com.br/costacurta/artma05_UC_Visaodehoje.htm (Acesso em: 09 fev. 2017).


[1] The multiple dimensions cited include: the forms of competition; technological advances, demographic trends, globalization of markets; emergence of new industries; changes in the workforce; supply chains, among others.


Received: April 24, 2017

Approved: April 31, 2017

DOI: 10.20985/1980-5160.2018.v13n4.1308

How to cite: Ramos, H. C.; Garcia, S. M. S.; Almeida, L. S. F. et al. (2018), “A documentary study on the evaluation of two result measurement indicators in corporate universities”, Sistemas & Gestão, Vol. 13, No. 4, pp. 424-432, available from: http://www.revistasg.uff.br/index.php/sg/article/view/1308 (access abbreviated month. year).



Creative Commons License
This work is licensed under a Creative Commons Attribution 4.0 International License.

 

ISSN: 1980-5160

Rua Passo da Pátria 156, bloco E, sala Sistemas & Gestão, Escola de Engenharia, São Domingos, Niterói, RJ, CEP: 24210-240

Tel.: (21) 2629-5616

Correspondência: Caixa Postal LATEC: 100175, CEP 24.020-971, Niterói, RJ