This article aims to demonstrate the application of the Analytic Hierarchy Process (AHP) method through a combination of relative and absolute measurement to treat a complex decision problem. The motivation for the use of this method came from the need of providing support to the Fluminense Federal Institute (IFF), a public institution that exercises teaching, research and extension activities, in the selection of several project propositions submitted to the research department. Such projects are submitted to an evaluation of different criteria which are mostly qualitative and difficult to measure. This is a case study in research area of the Fluminense Federal Institute, based on literature and document research and unstructured interviews. The application of the method allowed a more efficient evaluation process and generated a priority list of projects complying with the requirements for a scientific research project, to meet the institution's goals.
Keywords: multiple criteria, AHP, project selection.
Multi-criteria decision support methods (MDS) have been used in different areas of knowledge (Rodriguez et al., 2013). Such methods help decision-making processes in which there is evaluation of alternatives in multiple dimensions based on the preferences issued by decision makers (Gomes et al., 2004; Haimes et Chankong, 1979).
According to Vincke (1992), professionals and experts are increasingly aware of the need to incorporate multiple criteria of any kind in problems of management or decision, to meet the different existing dimensions in a problem. This fact demands tools to support decision-making that will help the decision maker to consider the alternatives at multiple points of view while seeking a satisfactory solution.
Such complexity is also observed in the problems regarding scientific research projects selection - specifically, the research area of the Fluminense Federal Institute (IFF), a federal government agency that operates in the higher, basic and vocational education sector. The IFF research area has as one of its tasks the management of granting incentive sponsorship for scientific research, due to the scarce distribution of resources to meet the increasing demand for research projects. The organism observed the need for a systematic method of selection of the projects which fit better to IFF institutional purposes. This is, therefore, a priority problem due to the importance of each project to the institution.
Francisco (2002) points out that the considerable increase seen in research activities involving limited resource allocation efforts has led both public and private research bodies to use systematic evaluation processes for the selection of research projects propositions. Such a system serves both to justify the choices of managers in relation to investment in research and to ensure to the public a good allocation of resources, allowing greater transparency in decision making.
Freitas (1998) mentions that evaluation, judgment and choosing are part of the normal behavior of human life, and play an even more important role in scientific activities. According to him, the scientists, private and public organizations use the assessment as a necessary tool to promote the development and improvement of their activities and businesses.
According to Lima (2008), the allocation of resources can be achieved through the optimization of resources or by seeking the preferential alignment of alternatives for prioritizing the ones more capable of meeting the organization's mission. The latter is the focus of this work. Keeney (1992) also emphasizes the relevance of considering the organizational goals in the choice of projects.
However, in literature, there was no evidence of MDS application to assist in the selection of scientific research projects in a Science and Technology Institute (STI). Thus, this paper proposes the use of a MDS method, the Analytic Hierarchy Process (AHP), which combines the relative measurement with absolute measurement (Saaty, 1990) for analysis and selection of a large number of scientific research project proposals for scholarships research purposes. As we will demonstrate in the study, the use of MDS method allows the selection of projects aligned with the institutional goals of IFF, thereby promoting a more efficient and judicious allocation of awarded grants.
The choice for AHP was motivated by its simplicity and easy implementation and also due to the qualitative nature of the problem presented in most of the variables. The combination with absolute measurement is justified by the possibility of handling a large number of alternatives.
Another reason for choosing this method is because it enables structuring the problem hierarchically, allowing us to represent a subordination relation between the elements for achieving the main objective. In addition, the AHP facilitates the measurement of qualitative elements and allows the sorting of the elements according to their contribution to the achievement of the desired goal.
The method is well suited to the research problem of this study, which consists in overcoming the difficulty for a Science and Technology Institute as the IFF of proposing a method that ensures the alignment of selected projects to the institution's objectives, considering a set of factors, especially the qualitative ones, to be satisfied. The AHP, by combining the relative and absolute criteria, aims to generate a ranking to identify the projects that contribute most satisfactorily for the purposes of the STI.
AHP is one of MDS methods widely used in the evaluation of multiple criteria and goals in problems characterized by their complexity and subjectivity (Shimizu, 2006; Shin et al, 2013). Created by Thomas L. Saaty in the 1970s, this method consists in the development of a model that reflects the workings of the human mind in the evaluation of the alternatives facing a complex decision problem.
Moreover, the method allows us to deal with problems involving both tangible and intangible values, thanks to its ability to create measures for qualitative variables based on subjective judgments made by the decision makers (Saaty, 1991). The AHP allows treating complex problems in a simple way (Costa et Moll, 1999).
According to Saaty (1991), the application of AHP comprises the following stages: structuring of criteria and alternatives; collecting judgments; calculating priorities; checking the consistency of judgments; and lastly, calculating the overall priorities of the alternatives. The structuring of the criteria consists in modeling the decision problem in a hierarchical structure, which, starting from the main goal is decomposed in the different criteria needed to achieve the referred goal, forming thus a layer of criteria. Each element of this layer, in turn, can be decomposed into two or more criteria, and so on, making easier the understanding and processing of the problem. Each element of the last level (children-element) is decomposed into alternatives, allowing an assessment in the perspective of each parent goal.
However, according to Miller (1956), we must use a maximum of 7 ± 2 elements in each level in order to achieve higher accuracy in the comparisons. Also, Alves et Alves (2015) concluded that a large number of comparisons may cause a risk of inconsistencies in the judgments. Saaty (1991) mentions that there is no standard procedure for selecting criteria and goals. The author suggests the use of brainstorming with experts and / or bibliographic research to assist in clarifying the criteria and goals.
According to Saaty (1991), following the hierarchy modeling of criteria comes their evaluation by the decision makers. These judgments are carried out through pairwise comparisons between two elements of the same level against the next element on a higher level. The elements are compared from a square matrix, whose order is equal to the number of elements subordinate to the next higher node. Subordinate elements are arranged in the same order, forming the rows and columns of the matrix.
According to that author, during the evaluation, each element in the line is compared with each column element and the score of the judgment is then registered in the matrix, in the row and column position regarding the compared elements. Table 1 shows the generic judgment matrix of n alternatives (a1, a2,..., an) compared to the criterion C1, where xij represents the judgment inputs i and j that range from 1 to n. When comparing the two elements the assessors must take into account which one is the most important element based on the focus criterion and the intensity of this importance.
As Saaty (1991) states, the comparison matrix generates reciprocal relations as shown in Table 1. Thus, for each judgment position recorded in line i and column j, represented by xij, there is a value equal to 1/xij in the reciprocal position, i.e., it is in column j, row i position. Considering the positions of line and column elements i and j which, respectively, range from 1 to n, the xij elements obey the following rules:
Rule 1: If xij = α, then xji = 1/α, α≠0, where α is the numerical value of the judgment based on Saaty’s scale (1991). So, xji = 1/xij.
Rule 2: If ai is judged as equally important as aj, then xij = 1 and xji = 1; and, in particular, xij = 1, ∀i=j
Table 1. Judgment Matrix
Source: Based on Saaty’s work (1991)
The judgment shall be based on the Saaty’s scale (1991) as shown in Table 2, aiming first to the conceptual judgment and then to conversion to a numerical scale in order to register the result, as well as its reciprocal associated evaluation, in the matrix. It is necessary to perform n (n-1) / 2 comparisons for the decision maker, as n is the number of compared elements (Gomes et al., 2004).
Table 2. Saaty’s Judgment Scale
Source: Based on Saaty’s work (1990)
The next stage is to calculate local and global priorities. In other words, it is to calculate the relative contribution of each element into hierarchical structure relative to the immediate goal and in relation to the main goal. First, the calculation regarding the priority of each element (node) in relation to its immediately upper element is made, and the local mean priority of the node is established. Then we calculate the global priority (regarding the main goal) of the respective element, multiplying its local average priority by the local average priorities of the hierarchically superior nodes (Vargas, 2010).
According to Saaty (1991), the local average priorities of elements compared in the judgment matrix can be obtained by matrix operations, calculating and normalizing matrix main eigenvector. However, that author presents other simpler procedures to generate the vector of priorities with approximate values. One of these procedures was used by Vargas (2010) and is adopted for this study, as follows: i) we calculate the sum of the judgments recorded in each column of the judgment matrix; ii) we create a new normalized matrix in which each element is initialized by the element of the original matrix divided by the total of its respective column; iii) we then calculate priority by the arithmetic mean of the elements of each row of the normalized matrix.
The result in each row is the total percentage of relative priorities or preferences in relation to the immediate goal focused. The resulting vector of priorities is called matrix eigenvector, and the sum of its elements is equal to 1. After the calculation of local priorities for each next higher node, we then calculate the consistency of such judgments.
When considering the intrinsic difficulties of the human being to make decisions on issues with too much information and multiple criteria, Saaty (1991) proposed a procedure to calculate inconsistencies derived from the value judgment between the elements compared in a complex decision problem. The author admits a 10% tolerance range for any inconsistencies. Vargas (2010) describes in a simple way the steps to check the consistency of judgment.
According to Vargas (2010), in the first step, we calculate the largest eigenvalue of the judgment matrix (λMax) by getting the sum of the product of each total in column j of the original judgment matrix by each element in the position j of the priority vector, considering j as the column of judgment matrix that ranges from 1 to n. Considering the judgment matrix, the priorities vector (the calculated priorities of the elements) and the order (n) of the matrix array, calculating the eigenvalue can be represented by the following formula:
(1)
Where Tj is the sum of the column j of judgment matrix and Pj is the calculated priority for the criterion located in row j.
In the second step, we calculate the Consistency Index (CI) as follows:
(2)
In the third step, we calculate the Consistency Ratio (CR):
(3)
According to Saaty (1991), the Random Index (RI) is the consistency index of a reciprocal matrix generated randomly by the Oak Ridge laboratory. Table 3 shows the RI table containing the random indexes calculated by the Oak Ridge laboratory for reciprocal square matrices of order n. As Saaty (1991) stated, if the CR calculated is less than or equal to 0.10, the judgment matrix is considered consistent. Otherwise, the array is considered inconsistent, and the judgment step must be redone.
Table 3. Random consistency index table
Source: Saaty (1991)
After checking the consistency of judgments, we calculate the overall performance of the alternatives. According to Saaty (1991), and based on the hierarchical structure of the AHP, the global priorities calculated for each criterion correspond to the importance of each criterion in relation to the main goal.
However, at alternatives level, the priority value found by multiplying the local priority of an alternative in relation to a specific focus by the global priority of this alternative reflects its impact regarding the main goal to a single criterion. Therefore, in order to obtain the global priority of alternatives, we must calculate the sum of the global priorities of the alternatives calculated for each criterion. This priority will determine the contribution of the alternative to the main goal.
According to Saaty (1990), absolute measurement is applied when it’s needed to measure the elements in each criterion based on a conceptual scale. First, we must establish the conceptual degrees as, for example, excellent, very good, good, average, regular and bad. After setting the nominal scale to be considered for each criterion, we do a pairwise comparison of the scales based on each criterion in order to obtain the weight of each related scale element by calculating the priorities. When assembling the matrix of judgment, the elements scale must be sorted from best to worst, arranging them in columns from right to left, and in lines from top to bottom, respectively.
The authors Padovani et al. (2010), Archer et Ghasemzadeh (1999) and Saaty (2005) indicate that the AHP is not feasible when dealing with a large number of alternatives. Saaty (2005) proposes the use of AHP combined with absolute measurement as a solution. The latter can be used to measure the alternatives for each criterion. The global value of alternative can be obtained through the sum of the product of overall weight of each criterion (obtained by relative measurement) by the evaluation of alternative obtained by the absolute measurement in the respective criteria (Saaty, 1990). Furthermore, according to Saaty (1987), the combination of these two measurements solves the problem of reversion order mentioned by Shin et al. (2013).
This section describes the steps of the application of AHP combining relative and absolute measurement to solve the problem of selection of research proposals at the Fluminense Federal Institute. The referred problem has a large number of alternatives submitted to an evaluation of multiple and mostly qualitative criteria.
Several evaluators participate in this, so there is a need for standardization and systematization of the work process. It is seen as a complex decision problem therefore. In addition, the proposed study aims to choose the projects most relevant to the institutional propositions, given that those start from the proposers’ idea. This allows better allocation of funding resources for research.
The AHP suited well to the problem due to its resources, such as the measurement of qualitative variables of the problem and the degree of contribution of each criterion to the main objective, the generation of the final ranking of alternatives, in order to allow managers choosing the best project proposals.
The relative measurement was applied to assess each criterion towards the main goal. The absolute measurement was used to evaluate each alternative in each aspect or sub-criterion. The overall performance of each alternative was obtained by the weighted sum which is the sum of the products of the relative measurement of criteria by the absolute measurement of the alternative of the respective criterion.
For the development of this study, we established a commission with five members, as follows: one author of this work and four Institution Research Department representatives. This committee was responsible for validating and approving actions and decisions taken by the authors in each stage of the work. This also helped in communication with other members of the research area, allowing their feedback and necessary adjustments during this work. The application of the model involved the stages described in the following subsections.
We identified the main criteria and established their aspects based on document research, unstructured interviews and literature review. It was found that, by nature, some criteria were related to institutional objectives while others to the quality of technical and scientific research.
Therefore, the restructuring of the criteria resulted in two groups, A and B, whose hierarchical structures are shown in Table 4 and Table 5 respectively, together with criteria identifiers assigned by the authors.
The two sets of criteria made it possible to establish a two-step selection stage. The group A aims to measure the contribution of the alternatives to meet institutional goals, while group B is designed to measure the contribution of the alternatives in relation to the requirements for each scientific research. Thus, the best alternatives in first stage serve as input to the second step. And this results in a more consistent selection.
Each criteria group was organized complying with the AHP hierarchy. Level 1 is the main focus. Level 2 represents the identified criteria layer. At level 3, aspects of each criterion are represented and modeled as sub-criteria.
Table 4. Criteria Group A
Source: Authors’ study
Table 5. Criteria Group B
Source: Authors’ study
3.2 Collecting of judgment criteria, calculation of priorities and consistency The process of measurement of criteria begins with the collection of judgment of each criterion relative importance by the immediate focus criterion. Thus, the judgment of subordinate criteria in relation to the next higher node was conducted for each group of criteria.
As a collection instrument, we used the AHP judgment matrix. The result of each pair of criteria compared was recorded directly in the judgment matrix by one of the authors according to the managers’ consensus.
The array was configured in electronic data sheet to receive the data reported above the diagonal positions and to calculate automatically the value of the reciprocal positions associated with each entry, facilitating the registration of judgments. The example in Table 6 shows the result of the collection of judgment criteria for the level 2 criteria in group A in relation to the focus criterion.
Table 6. Judgment of the importance of A1-A4 criteria in relation to the A main goal.
Source: Authors’ study
The judgment matrix was normalized for the calculation of the criteria priorities. We then calculated the arithmetic mean of the elements in each row to get the average local priority. The local priority of each element of the level 2 has the same value as the global priority, and its value can be represented in percentage format, demonstrating the degree of their contribution to the achievement of the main objective. The weight of the main objective is 1, i.e., it’s equal to the sum of all the weights (priorities). Table 7, below, shows the normalization performed and the priorities calculated based on the judgment seen in Table 6. All the calculation was integrated into the judgment matrix in a way that, when registering the trials, the priorities could be calculated automatically.
Calculations to check the consistency of the trial were also implemented in the data spreadsheet and integrated to the judgment matrix to calculate automatically the ratio of consistency. So for the record of the judgment, managers, based on the calculated CR, could see their decisions and seek a consensus again in case of any inconsistency. Intermediate values considered in CR calculation based on the exposed judgment matrix in Table 6 are demonstrated in Table 8.
Table 7. Calculated priorities of the criteria judged in relation to main goal A Normalization
Source: Authors’ study
Table 8. Intermediate values considered in the calculation of the consistency ratio of the judgment of A1-A4 criteria in relation to main goal A
Source: Authors’ study
Then the level 3 criteria (aspects) were measured. In this case, we considered as the focus criterion for each aspect its immediately higher hierarchical level 2 element. Again, the collection of judgment and calculation of priorities of the aspects were carried out, and it was the consistency of judgments was analyzed with the use of the focus criterion. In this case, the global priority of each aspect was obtained calculating the product of global priority of the focus node by the aspect local priority. The sum of the sub-criteria local priorities must be equal to the global priority of the focus criterion. All calculations were set in a data sheet file in order to allow the integration of the priorities generated in each node of the hierarchy. Tables 9, 10 and 11 show, respectively, the collection of judgment of A1.1-A1.3 sub-criteria in relation to the criterion A1 (scientific and technological advance), the average local priorities, the calculated global priorities and the consistency ratio regarding the judgment.
Table 9 – Judgment of A1.1-A1.3 criteria in relation to criterion A1
Source: Authors’ study
Table 10 – Priority of A1.1-A1.3 criteria in relation to criterion A1 Normalizing
Source: Authors’ study
Table 11. Intermediate values considered in the calculation of the consistency ratio of the judgment of A1.1-A1.3 criteria in relation to criterion A1
Source: Authors’ study
Table 12 and 13 show the measurements of criteria groups A and B respectively.
Source: Authors’ study
Table 13. Priorities (weights) generated by AHP for criteria and aspects in group B
Source: Authors’ study
As we can observe, most of the judgments presented consistency ratio equal or close to zero. This demonstrates a greater consistency in the judgment of the criteria, which can enhance the quality of the managers’ decision making process.
Based on the criteria and aspects, two evaluation sheets were developed, one for each set of criteria. Each card assessment items were textually represented by the aspects of their respective criteria structure. Thus, the weight of each evaluation item is obtained by its overall weight as previously generated by AHP.
In each form, we defined for each assessment item the conceptual scale of values to be used by project evaluators in assessing each aspect. The use of conceptual scale facilitated the evaluation because, by nature, most of the aspects were qualitative. For some quantitative aspects, conversion scales were established, respecting the restrictions imposed by the context. Such scales have been defined by the authors and the project evaluators committee. The assessment items that make up each evaluation form and its nominal scales, used as conceptual answers, can be seen in Table 14 and Table 15 respectively, as follows.
After setting the scales, we proceeded again to collect the judgment of the degree of importance of the nominal scales in relation to the associated aspect. The calculation of priorities and consistency ratio was automatically done by the spreadsheet, so that at each record of judgment, one could realize the degree of consistency in the conducted judgment. The priority found for each nominal scale is used to automatically measure the response of the evaluator at the moment of project assessment. Table 14 and Table 15 show, for each group of criteria, the aspects (Evaluation items) and their scales (conceptual value of answers) along with their local priorities (LP) generated by the AHP and the value of the CR resulting from each judgment.
Table 14. Weights of nominal scales used in the assessment based on the criteria group A
Source: Authors’ study
Table 15. Weights of nominal scales used in the assessment based on the criteria group B
Source: Authors’ study
We observe that all trials presented consistency ratio within the limits set by Saaty (1991), as most presented zero CR value. Therefore, all the trials performed demonstrated consistency.
Collecting the evaluation of each project was carried out by means of evaluation forms. These were sent to the evaluators, whose answers were duly recorded in the data sheets. These spreadsheets were previously configured in a way that, when registering the obtained conceptual evaluations of projects, it could automatically assign the weights and then calculate the performance of projects.
Table 16 exemplifies the calculation of the performance of a given project relative to its alignment with the goal of the institution, based on the criteria group A. The same calculation was made based on the responses found in the evaluation of that project in relation to the criteria group B. It was observed that some projects had matching performances. Six decimal places were applied on project global performance indicators in order to minimize the coincidences.
Table 16. Performance calculation of Project 1 based on criteria group A
Source: Authors’ study
At this stage, we performed the ordination of projects in descending order of score as obtained in the evaluation of criteria related to the institutional goals, the criteria group A. This is shown in Table 17 below.
Table 17. Project ranking according to its contribution to the achievement of institutional goals
Source: Authors’ study
The projects that achieved the best scores in the first ranking were kept in the research evaluation spreadsheet while the others were excluded. Then we proceeded to the ordination of projects according to their performance data in relation to the assessment based on criteria group B. The result is shown in Table 18 as follows.
Table 18. Project ranking according to its compliance with scientific research requirements
Source: Authors’ study
Ranking final results pointed the projects that meet both the quality of research and institutional interests. This ranking served as a support resource for the managers to the allocation of research grants.
The combination of AHP relative and absolute measurements simplified and facilitated the evaluation. The nominal scales used in the absolute measurement allowed a conceptual evaluation, favoring the assessment of the qualitative items.
In addition, the process of modeling the problem in AHP led managers to reflect and discuss about the contribution of each criterion considered for achieving the institutional goals and also the quality of each research proposed. The restructuring was important in the standardization of aspects considered in each criterion, allowing to everyone involved in the process a common understanding and greater consistency in the evaluation.
The final ranking, generated from filtering results in the first ranking, secured the choice of the projects most compliant to research and institution goals. Since the institution's goals are in line with the needs of society and its local and regional development, the method contributed to improving quality in a decision making process.
The incorporation of managers’ preferences based on a consensus model favored the compromise between the parties, thereby increasing the credibility of the decision-making process, which results in a more consistent and effective decision model.
Despite the satisfactory result, we observed that the ranking results produced some similar or close scores, because some projects had coincidentally the same evaluation values.
In addition, some managers wanted to categorize projects as good, bad and very bad. However, as the proposed here was the prioritization of projects, we recommend as future work a study about the feasibility of implementing the AHP combined with other methods for the final categorization of projects, in order to analyze the effectiveness of the proposed solution in relation to others from different multi-criteria selection models.
Alves, J. R. X. et Alves, J. M. (2015), “Definição de localidade para instalação industrial com o apoio do método de análise hierárquica (AHP)”, Production, Vol.25 No.1, pp.13-26, disponível em: http://dx.doi.org/10.1590/S0103-65132014005000023 (Acesso em 18 de abril de 2015)
Archer, N. P. et Ghasemzadeh, F. (1999), “An integrated framework for project portfolio selection”, International Journal of Project Management, Vol.17 No.4, pp.207-216, disponível em: http://www.scopus.com. (Acesso em 15 de outubro de 2012).
Costa, H.G. et Moll, R.N. (1999), “Emprego do método de análise hierárquica (AHP) na seleção de variedades para o plantio de cana-de-açúcar”, Gestão & Produção, Vol.6 No.3, pp.243-256, disponível em: http://www.scielo.br (Acesso em: 08 de agosto de 2013).
Francisco, L. T. S. T. (2002), Indicadores para avaliação de resultados de projetos de pesquisa científica e tecnológica. Dissertação de Mestrado em Administração, Universidade Federal do Rio Grande do Sul, Porto Alegre.
Freitas, M. H. A. (1998), “Avaliação da produção científica: considerações sobre alguns critérios”, Psicol. Esc. Educ. (Impr.), Vol. 2 No. 3, pp.211-228, disponível em http://www.scielo.br (Acesso em: 20 de dezembro de 2013).
Gomes, L. F. A. M., Araya, M. C. G. et Carignano C. (2004), Tomada de Decisões em Cenários Complexos, Tradução de Marcela Cecilia González Araya, Pioneira Thomson Learning, São Paulo, SP.
Haimes, Y. Y. et Chankong, V. (1979), “Kuhn-Tucker Multipliers as Trade-Offs in Multiobjective Decision-Making Analysis”, Automatica, Vol.15, pp.50-72, disponível em: http://www.scopus.com (Acesso em: 05 de julho de 2013).
Keeney, R. L. (1992), Value-focused thinking a path to creative decision making. Harvard University Press, Cambridge.
Lima, A. S. de. (2008), Proposta de método para modelagem de critérios de priorização de projetos de pesquisa e desenvolvimento aeroespaciais. Dissertação de Mestrado em Engenharia Aeronáutica e Mecânica, Instituto Tecnológico de Aeronáutica, São José dos Campos.
Miller, G.A. (1956), “The magical number seven, plus or minus two: Some limits on our capacity for processing information”, Psychological review, Vol.63 No.2, pp.81-97, disponível em http://www.scopus.com (Acesso em 01 de novembro de 2012).
Padovani, M., Carvalho, M. M. de et Muscat, A. R. N. (2010), “Seleção e alocação de recursos em portfólio de projetos: estudo de caso no setor químico”, Gest. Prod., Vol.17 No.1, disponível em: http://www.scielo.br/scielo.php?script=sci_arttext&pid=S0104-530X2010000100013&lng=pt&nrm=iso (Acesso em: 6 de março de 2014).
Rodriguez, D. S. S, Costa, H. G. et Carmo, L. F. R. R. S. do (2013), “Métodos de auxílio multicritério à decisão aplicados a problemas de PCP: Mapeamento da produção em periódicos publicados no Brasil”, Gest. Prod., Vol.20 No.1, pp134-146, disponível em: http://www.scielo.br/pdf/gp/v20n1/a10v20n1.pdf (Acesso: 11 de fevereiro de 2015).
Saaty, T.L. (1987), “Rank generation, preservation, and reversal int the analytic hierarchy decision process”, Decision Sciences, Vol.18, No.2, pp.157-177, disponível em: http://onlinelibrary.wiley.com/doi/10.1111/j.1540-5915.1987.tb01514.x/abstract (Acesso em: 18 de abril de 2015).
Saaty, T.L. (1990), “How to make a decision: The analytic hierarchy process”, European Journal of Operational Research, Vol.48 No.1, pp.9-26, disponível em: http://www.scopus.com (Acesso em: 12 de outubro de 2012).
Saaty, T. L. (1991), “Método de Análise Hierárquica”, Tradução de Wainer da Silveira e Silva, McGraw-Hill, Makron, São Paulo, SP.
Saaty, T. L. (2005) “The Analytic Hierarchy and Analytic Network Processes for the measurement of intangible criteria and for decision-making”, em Figueira, J., Greco S. e Ehrgott, M. Multiple Criteria Decision Analysis: state of the art surveys, Springer Sciense+Business Media, New York, pp.345-408.
Shimizu, T. (2006), Decisão nas Organizações. 2 ed., Atlas, São Paulo, SP.
Shin, Y.B., Lee, S., Shun, S.G. et Chung, D. (2013), “A critical review of popular multi-criteria decision making methodologies”, Issues in Information Systems, Vol.14 No.1, pp.358-365, disponível em http://iacis.org/iis/2013/225_iis_2013_358-365.pdf (Acesso em 14 de abril de 2015).
Vargas, R.V. (2010), “Utilizando a programação multicritério (Analytic Hierarchy Process – AHP) para selecionar e priorizar projetos na gestão de portfóliao”, em PMI Global Congress – América do Norte, 2010, Washington – EUA, disponível em http://www.ricardo-vargas.com/articles/analytic-hierarchy-process/#portuguese (Acesso em 10 de outubro de 2012).
Vincke, P. (1992), Multicriteria decision-aid, John Wiley & Sons Ltd., England.