First, read the article on "The Delphi Method for Graduate Research." ------ Article is posted below

Include each of the following in your answer (if applicable – explain in a paragraph)

Research problem: what do you want to solve using Delphi?

Sample: who will participate and why? (answer in 5 -10 sentences)

Round one questionnaire: include 5 hypothetical questions you would like to ask

Discuss: what are possible outcomes of the findings from your study? Hint: this is the conclusion. (one paragraph/5-7 sentences)

Your hypothetical Delphi study should address a research problem and provide explanation about each step in the process. You should provide examples of questions you would ask, and potential benefits/outcomes of the study.


The Delphi Method for Graduate Research Gregory J. Skulmoski Zayed University, Dubai, United Arab Emirates [email protected] Francis T. Hartman and Jennifer Krahn University of Calgary, Calgary, Canada [email protected] [email protected] Executive Summary The Delphi method is an attractive method for graduate students completing masters and PhD level research. It is a flexible research technique that has been successfully used in our program at the University of Calgary to explore new concepts within and outside of the information systems body of knowledge. The Delphi method is an iterative process to collect and distill the anonymous judgments of experts using a series of data collection and analysis techniques interspersed with feedback. The Delphi method is well suited as a research instrument when there is incomplete knowledge about a problem or phenomenon; however it is not a method for all types of IS research questions. The Delphi method works especially well when the goal is to improve our understanding of problems, opportunities, solutions, or to develop forecasts. In this paper, we provide a brief background of the Classical Delphi followed by a presentation of how it has evolved into a flexible research method appropriate for a wide variety of IS research projects, such as determining the criteria for IS prototyping decisions, ranking technology management issues in new product development projects, and developing a descriptive framework of knowledge manipulation activities. To illustrate the method’s flexibility, we summarize distinctive non-IS, IS, and graduate studies Delphi research projects. We end by discussing what we have learned from using the Delphi method in our own research regarding this method's design factors and how it may be applied to those conducting graduate studies research: i) methodological choices such as a qualitative, quantitative or mixed methods approach; ii) initial question degree of focus whether it be broad or narrowly focused; iii) expertise criteria such as technical knowledge and experience, capacity and willingness to participate, sufficient time, and communication skills; vi) number of participants in the heterogeneous or homogeneous sample, v) number of Delphi rounds varying from one to 6, vi) mode of interaction such as through email, online surveys or groupware, vii) methodological rigor and a research audit trail, viii) results analysis, ix) further verification through triangulation or with another sample, and x) publishing of the results. We include an extensive bibliography and an appendix with a wide-ranging list of dissertations that have used the Delphi method (including brief research description, number of rounds and sample size). The Delphi method is a flexible, effective and efficient research method that can be successful used by IS graduate students to answer research questions in information systems and to rigorously advance the IS body of knowledge. Material published as part of this publication, either on-line or in print, is copyrighted by the Informing Science Institute. Permission to make digital or paper copy of part or all of these works for personal or classroom use is granted without fee provided that the copies are not made or distributed for profit or commercial advantage AND that copies 1) bear this notice in full and 2) give the full citation on the first page. It is permissible to abstract these works so long as credit is given. To copy in all other cases or to republish or to post on a server or to redistribute to lists requires specific permission and payment of a fee. Contact [email protected] to request redistribution permission. The Delphi Method for Graduate Research 2 Keywords: Graduate studies, Delphi Method, qualitative research, quantitative research, questionnaire surveys. Introduction It continues to be an exciting time to be a researcher in the information systems discipline; there seems to be a plethora of interesting and pressing research topics suitable for research at the masters or PhD level. Researchers may want to look forward to see what will be the key information systems issues in a wireless world, the ethical dilemmas in social network analysis, and the lessons early adopters learn. Practitioners may be interested in what others think about the strengths and weaknesses of an existing information system, or the effectiveness of a newly implemented information system. The Delphi method can help to uncover data in these research directions. The Delphi method is an iterative process used to collect and distill the judgments of experts using a series of questionnaires interspersed with feedback. The questionnaires are designed to focus on problems, opportunities, solutions, or forecasts. Each subsequent questionnaire is developed based on the results of the previous questionnaire. The process stops when the research question is answered: for example, when consensus is reached, theoretical saturation is achieved, or when sufficient information has been exchanged. The Delphi method has its origins in the American business community, and has since been widely accepted throughout the world in many industry sectors including health care, defense, business, education, information technology, transportation and engineering. The Delphi method’s flexibility is evident in how it has been used. It is a method for structuring a group communication process to facilitate group problem solving and to structure models (Linstone & Turloff, 1975). The method can also be used as a judgment, decision-aiding or forecasting tool (Rowe & Wright, 1999), and can be applied to program planning and administration (Delbeq, Van de Ven, & Gustafson, 1975). The Delphi method can be used when there is incomplete knowledge about a problem or phenomena (Adler & Ziglio, 1996; Delbeq et al., 1975). The method can be applied to problems that do not lend themselves to precise analytical techniques but rather could benefit from the subjective judgments of individuals on a collective basis (Adler & Ziglio, 1996) and to focus their collective human intelligence on the problem at hand (Linstone & Turloff, 1975). Also, the Delphi is used to investigate what does not yet exist (Czinkota & Ronkainen, 1997; Halal, Kull, & Leffmann, 1997; Skulmoski & Hartman 2002). The Delphi method is a mature and a very adaptable research method used in many research arenas by researchers across the globe. To better understand its diversity in application, one needs to consider the origins of the Delphi method. The Classical Delphi The original Delphi method was developed by Norman Dalkey of the RAND Corporation in the 1950’s for a U.S. sponsored military project. Dalkey states that the goal of the project was “to solicit expert opinion to the selection, from the point of view of a Soviet strategic planner, of an optimal U.S. industrial target system and to the estimation of the number of A-bombs required to reduce the munitions output by a prescribed amount,” (Dalkey & Helmer, 1963, p. 458). Rowe and Wright (1999) characterize the classical Delphi method by four key features: 1. Anonymity of Delphi participants: allows the participants to freely express their opinions without undue social pressures to conform from others in the group. Decisions are evaluated on their merit, rather than who has proposed the idea. 2. Iteration: allows the participants to refine their views in light of the progress of the group’s work from round to round. Skulmoski, Hartman, & Krahn 3 3. Controlled feedback: informs the participants of the other participant’s perspectives, and provides the opportunity for Delphi participants to clarify or change their views. 4. Statistical aggregation of group response: allows for a quantitative analysis and interpretation of data. Some (Rowe & Wright, 1999) suggest that only those studies true to their origins that have the four characteristics should be classified as Delphi studies, while others (Adler & Ziglio, 1996; Delbeq et al., 1975; Linstone & Turloff, 1975) show that the technique can be effectively modified to meet the needs of the given study. Perhaps a distinction might be made by using the term Classical Delphi to describe a type of method that adheres to the characteristics of the original Delphi as summarized by Rowe and Wright (1999). Typical Delphi Process The Delphi process has been comprehensively reviewed elsewhere (Adler & Ziglio, 1996; Delbeq et al., 1975; Linstone & Turloff, 1975), and so we present only a brief overview of how we have used the Delphi in some of our graduate students' research projects (Figure 1). Figure 1: Three Round Delphi Process 1) Develop the Research Question - The research question is derived by a number of ways. For example, it might be co-developed by the student with the help of the supervisor, or the researcher’s own industry experience often contributes to his interest in the research area. A review of the literature is also conducted, among other things, to determine if a theoretical gap exists. Often pilot studies are undertaken for numerous reasons: i) identify the problem, ii) conceptualize the study, iii) design the study, iv) develop the sample, v) refine the research instrument, and, vi) develop and test data analysis techniques (Prescott & Soeken, 1989). Completing a pilot study can also help ascertain the relevance the research question has to industry; some supervisors strongly favor applied rather theoretical research. 2) Design the Research - After developing a feasible research question, we begin designing the research from a macro to a micro perspective. Typically we review different research methods (both qualitative and quantitative) and after considering the pros and cons of each, we select the most promising method(s) to help answer our research question. The researcher would select the Delphi method when he wants to collect the judgments of experts in a group decision making setting. Both qualitative and quantitative methods can be used in the Delphi process. The Delphi method may be only one component of the research project; for example, the Delphi outputs may be verified and generalized with a survey. 3) Research Sample - Selecting research participants is a critical component of Delphi research since it is their expert opinions upon which the output of the Delphi is based (Ashton 1986; BolPilot Studies Experience Literature Review Research Question Research Design Delphi R1 Design Delphi R1 Pilot Delphi R1 Survey & Analysis Delphi R2 Design Delphi R2 Survey & Analysis Delphi R3 Design Delphi R3 Survey & Analysis Research Documentation, Verification & Generalization Research Sample Pilot Studies Experience Literature Review Research Question Research Design Delphi R1 Design Delphi R1 Pilot Delphi R1 Survey & Analysis Delphi R2 Design Delphi R2 Survey & Analysis Delphi R3 Design Delphi R3 Survey & Analysis Research Documentation, Verification & Generalization Research Sample The Delphi Method for Graduate Research 4 ger & Wright 1994; Parente, Anderson, Myers, & O’Brien, 1994). There are four requirements for “expertise”: i) knowledge and experience with the issues under investigation; ii) capacity and willingness to participate; iii) sufficient time to participate in the Delphi; and, iv) effective communication skills (Adler & Ziglio 1996). Since expert opinion is sought, a purposive sample is necessary where people are selected not to represent the general population, rather their expert ability to answer the research questions (Fink & Kosecoff 1985). The student may need some help from the supervisor to identify the initial group of experts but may use the "snowball" sampling technique to generate subsequent participants (Hartman & Baldwin, 1995; Mason, 1996). 4) Develop Delphi Round One Questionnaire - Care and attention needs to be devoted to developing the initial broad question which is the focus of the Delphi because if respondents do not understand the question, they may provide inappropriate answers and/or become frustrated(Delbeq et al., 1975). Sometimes, the purpose of the first round Delphi is to brainstorm (R. Schmidt, 1997). 5) Delphi Pilot Study - A pilot study is sometimes conducted with the goals of testing and adjusting the Delphi questionnaire to improve comprehension, and to work out any procedural problems. The researcher may also pre-test each subsequent questionnaire. The Delphi pilot is especially important for inexperienced researchers who may be overly ambitious regarding the scope of their research or underestimate the time it will take a Delphi research participant to fully respond to the Delphi survey. 6) Release and Analyse Round One Questionnaire - The questionnaires are distributed to the Delphi participants, who complete and return them to the researcher. The results of Round One are then analysed according to the research paradigm (e.g. qualitative coding or statistical summarizing into medians plus upper and lower quartiles). Reality Maps can also be developed and shared with the Delphi participants. Reality Maps are graphical representations of the key constructs under investigation. They depict reality from the participant’s perspective and often illustrate interactions, causes and effects, process flow, and other aspects of their reality. Reality Maps can greatly improve understanding and facilitate the emergence of collective intelligence in subsequent rounds about the topic under investigation (Lindstone & Turloff, 1975). 7) Develop Round Two Questionnaire - The Round One responses are the basis with which to develop the questions in the Round Two Questionnaire. Depending upon the research goals, the researcher may direct the focus of the research, or be directed by the opinions of the participants. If the purpose of Round One was to generate a list, then it is common to pare down that list in Round Two (R. Schmidt, 1997). 8) Release and Analyse Round Two Questionnaire - The Round Two Questionnaire is released to the research participants and when completed, returned for analysis. However, the participants are first given the opportunity to verify that the Round One responses did indeed reflect their opinions and are given the opportunity to change or expand their Round One responses now that the other research participant’s answers are shared with them. Ranking and rating the output of the first round is common (R. Schmidt, 1997). Continuous verification throughout the Delphi process is critical to improve the reliability of the results (Adler & Ziglio, 1996; Delbeq et al., 1975; Linstone & Turloff, 1975) and should be factored into the research design. Again, a similar process of analysis is often used in Round Two. 9) Develop Round Three Questionnaire - The Round Two responses are used to develop the Round Three Questionnaire with additional questions to verify the results, to understand the boundaries of the research, and to understand where these results can be extended. Typically, the questions become more focused on the specifics of the research at each round. Skulmoski, Hartman, & Krahn 5 10) Release and Analyse Round Three Questionnaire - The final round of analysis is conducted following a similar process used to analyse the data in Rounds One and Two: use the appropriate technique for the question type (e.g. coding for open-ended, qualitative questions). Again, the research participants are given the opportunity to change their answers and to comment on the emerging and collective perspective of the research participants. The process stops if the research question is answered: for example, consensus is reached, theoretical saturation is achieved, or sufficient information has been exchanged. 11) Verify, Generalize and Document Research Results - The Delphi results are verified (usually continuously through the Delphi) and the extent the results can be generalized are also investigated. For PhD research conducted in our program, the Delphi results are often extended with a subsequent research phase such as interviews or surveys. The dissertation and thesis are sent to the National Library of Canada for their collection. These graduate students are also encouraged to publish their results in top tiered publications. The “typical” Delphi process that we follow in the Project Management Specialization Programme is a general guide rather than a template. That is, we modify the process to best answer our research questions. For example, different types of questions (closed/open) and analysis (qualitative/quantitative) can be used in each round. In our program we have used a three round qualitative Delphi process to develop a complexity-based project classification system (Skulmoski & Hartman, 2002). Other three round Delphi research projects were undertaken to identify and describe project manager fears and frustrations (Hartman & Jugdev, 1998), to identify the cost of mistrust in contracts (Zaghloul & Hartman, 2003) and to examine the fit between project manager leadership competencies and project characteristics (Krahn & Hartman, 2004). We have also used groupware technology to facilitate the Delphi method where only one round was required to achieve consensus regarding a new contracting process(Hartman & Baldwin, 1995). As groupware and distributed groupware become more widely available, the increase of single round Delphi studies may increase. How Others Have Used the Delphi Method Many have examined a variety of studies that have used the Delphi method (Adler & Ziglio, 1996; Linstone & Turloff, 1975; Rowe & Wright, 1999). The range of Delphi possibilities can be seen in Table 1. The Delphi has been used in research to develop, identify, forecast and to validate in a wide variety of research areas. While a three round Delphi is typical, single and double round Delphi studies have also been completed. Finally, the sample size varies in their studies from 4 to 171 "experts". One quickly concludes that there is no “typical” Delphi; rather that the method is modified to suit the circumstances and research question. Delphi Method Flexibility in IS/IT Research IS/IT researchers have also used the Delphi method. For example, the Delphi method has been used to select IS projects (Peffers & Tuunanen, 2005), specify IS project requirements (Perez & Schueler, 1982), to determine the criteria for IS prototyping decisions (Doke & Swanson, 1995), rank technology management issues in new product development projects (Scott, 2000), and to develop a descriptive framework of knowledge manipulation activities (Holsapple & Joshi, 2002). The Delphi method was used at the InSITE 2005 conference to identify topics that should be in an IT curriculum (Lunt et al., 2005). Next we will look at other IS projects that used the Delphi method in greater depth in order to both learn from the experiences of other researchers, and to display the flexibility of the method. Again, their focus, number of rounds and sample size are varied (Table 1). The Delphi Method for Graduate Research 6 Table 1: Delphi Method Diversity - Published Research Non IS/IT Study Delphi Focus Rounds Sample Size Gustafson, Shukla, Delbecq, & Walster (1973) Estimate almanac events to investigate Delphi accuracy. 2 4 Hartman & Baldwin (1995) Validate research outcomes. 1 62 Czinkota & Ronkainen (1997) Impact analysis of changes to the International business environment. 3 34 Kuo & Yu (1999) Identify national park selection criteria. 1 28 Nambisan et al.(1999) Develop a taxonomy of organizational mechanisms. 3 6 Lam, Petri, & Smith (2000) Develop rules for a ceramic casting process. 3 3 Roberson, Collins, & Oreg (2005) Examine and explain how recruitment message specificity influences job seeker attraction to organizations. 2 171 IS/IT Study Delphi Focus Rounds Sample Size Niederman, Brancheau, & Wetherbe, (1991) Survey senior IS executives to determine the most critical IS issues for the 1990s. 3 114, 126 & 104 Duncan (1995) Identify and rank the critical elements of IS infrastructure flexibility. 2 21 Brancheau, Janz, & Wetherbe (1996) Survey SIM members to determine the most critical IS issues for the near future. 3 78, 87 & 76 Nambisan et al. (1999) Develop a taxonomy of knowledge creation mechanisms. 3 11 Scott (2000) Rank technology management issues in new product development projects 3 20 Wynekoop & Walz (2000) Rank the most important characteristics of high performing IT personnel. 3 9 R. Schmidt, Lyytinen, Keil, & Cule (2001) Identify and rank software development project risks: an international comparative study. 3 Finland: 13, 13, & 13 Hong Kong: 11, 11 & 9 USA 21, 21 & 9 Keil, Tiwana, & Bush (2002) Rank software development project risks. 3 15, 15 & 10 Brungs & Jamieson (2005) Identify and rank computer forensics legal issues. 3 11 The Delphi method has been used to develop a taxonomy of knowledge creation mechanisms (Nambisan, Agarwal, & Tanniru, 1999). The researchers identified 19 knowledge creation Skulmoski, Hartman, & Krahn 7 mechanisms in the literature (e.g. IT journals, vendor demonstrations, IT steering committees, and user groups). They argue that deliberate organizational design in the form of mechanisms can facilitate user IT innovation to the benefit of the organization. Using a 3 round Delphi with 11 participants, the researchers populated a taxonomy of knowledge creation mechanisms. The participants were practicing IS managers from 6 organizations. They further verified the taxonomy in a field study using both the interview and survey methods. While this study is rigorous, we are unable to assess the "expertise" of the sample. The increasing reach and range of computers into society has both positive and negative effects. One of the insidious aspects of this adoption is e-crime. The extraction and presentation of electronic evidence in the courts form an important and new area of computer forensics. However, there are many emerging and difficult legal issues to address. The Delphi method was used by a team of researchers to identify the principle legal issues facing the computer forensics discipline within the Australian context (Brungs & Jamieson, 2005). A difficulty with this type of research is that there so few recognized experts in this field. Of these 30 experts, 11 participated in a three round Delphi study. This heterogeneous sample comprised of three distinct groups: police, regulators and consultants. The sample identified 17 issues in a brainstorming session, then they ranked and rated these in the Delphi. Selecting, implementing and using an IT infrastructure is a critical process for organizations to achieve their organizational goals. This task is complicated when goals and priorities change. A flexible IT infrastructure is therefore desirable. One researcher used the Delphi method to identify the characteristics and metrics of a flexible IT infrastructure (Duncan, 1995). Duncan used a two round Delphi (survey and discussion) to answer her research question. A homogeneous group of 21 participants from the senior ranks of Fortune 500 companies participated. In the first Delphi round, the participants rated flexibility characteristics (e.g. compatibility rules for communication networks, data and applications, management leadership in long term planning for applications, and interface standardization) that were identified in a literature review. Participants were also afforded the opportunity to add characteristics not on the initial list. In the second round, they discussed the round 1 results. The Delphi was followed by a round of interviews with a different sample for verification and generalization purposes. The Delphi method was used to investigate the traits and behaviors of top performing software developers (Wynekoop & Walz, 2000). They extended a previous study that used the interview method and choose the Delphi method because of its ability to achieve consensus; something that was absent among their interview sample. In this pilot study, nine participants (MBA students) were involved in a 3 round Delphi. She correctly identifies a limitation of her sample and alludes that while they have software development experience, they are not experts. Researchers have used the Delphi to identify software development project risks (Keil et al., 2002; R. Schmidt et al., 2001). In the first study, the focus was on developing a list of common risk factors in three settings: Hong Kong, Finland and the United States (R. Schmidt et al., 2001). In the second study, the researchers investigated the differences of opinion regarding software development project risks between users and the project manager using risks identified in the Schmidt study (Keil et al., 2002). In the Schmidt study, participants from all three countries participated in a brainstorming session to identify software project risks. In the subsequent rounds, they were divided according to their country. In the second round, they pared down the list, and in the final round, they ranked the risks. The researchers calculated the mean rank for each risk, and the degree of consensus within each country using Kendall's W. This study was extended by Keil's research team using the same Delphi method (three rounds, 15 participants) and analysis to reconcile user and project manager perceptions of risk. The Delphi Method for Graduate Research 8 Finally, the Delphi method has been used to forecast key issues in IS management. A 3 round Delphi was used where the Society of Information Systems (SIM) members were asked to rank IS issues (Niederman et al., 1991). The remarkable aspect of this study is that of the 241 Delphi surveys that were distributed, 114, 126 and 104 surveys were returned and usable over the three rounds. A similar study six years later also used the Delphi and SIM members to rank 21 MIS issues. Once again, the number of participants was high (but the response rate was lower) where of the 217 surveys that were sent out, 78, 87 and 76 surveys were returned and usable over the three rounds. There were significant changes in the rankings of the issues: for example, developing an information architecture was ranked the number one issue in the first study and dropped in the ranking in the second study to fourth place. The sixth ranked issue – building a responsive IT architecture – in the first study was ranked as the top issue in the second study. A discussion of the movement in the rankings is absent in the second study. The Delphi method has been used on different occasions in IS research. There is also wide variance in the sample size. Both heterogeneous and homogeneous samples were used. The degree of expertise of the sample also varied. Some studies began with a predetermined list to rank and rate, while others generated the initial list through brainstorming. However, three round Delphis appear to be favored. Some studies employed statistical treatment of data such as the Kendall W procedure. Most verified their research with another method, and if they did not they cautioned the reader when interpreting the results. While it is a flexible method, it has not seen the degree of use as the survey method. We believe that the Delphi can be an effective and efficient method appropriate for some IS research if rigorous design considerations are followed and implemented. The Delphi Method in Dissertations We see similar flexibility in the way the Delphi method was used in doctoral and masters research projects as before. Oddly, however, there are few research projects that have used the Delphi method identified in the literature review. Instead other methods used in dissertation and thesis projects such as surveys or interviews greatly outnumbered the Delphi. There are notable recent exceptions including: • Identifying the critical success factors for ERP implementation projects (Carson, 2005); • Developing a model of how technologies are developing and how they may fit with an organizational strategy (Gerdsri, 2005); • Improving the quality of IT security audits (Pieko, 2005); • Identifying the criteria for measuring knowledge management efforts (Anantatmula, 2004); • Identifying why the strategies for a Defense Department IT project succeeded or failed (Birdsall, 2004); and, • Identifying emerging IT issues of the 21st century that affect public school board policies (Dahlby, 2004). These dissertations reveal the variety of research questions in IS that can be asked and subsequently answered using the Delphi method. Approximately 40 dissertations and two theses that used the Delphi method were examined (See Appendix). Indeed, a search through the ProQuest Digital Dissertations database reveals at least 280 dissertations and theses that used the Delphi method in their research. The majority of the research projects were from either education or healthcare. Beginning with the initial Delphi question(s) in round 1, they can be either broad or narrow. Many (Alexander, 2004; Christian, 2003; Good, 1998) began with open questions in round 1 while some (Ayers, 1985; Friend, 2001; Menix, 1997) used narrow questions that focused on literature derived content. Skulmoski, Hartman, & Krahn 9 Sampling in these graduate student research projects also mirror other Delphi projects (See Appendix). We continue to see great variability in the number of participants from 8 (Friend, 2001) to 345 (Lecklitner, 1984). Here, such a large number of research participants was in part due to a heterogeneous sample with six distinct sub-groups. Others also used heterogeneous samples (Cabaniss, 2001; Menix, 1997; Rosenbaum, 1985) while the majority relied upon homogeneous samples. These were purposive samples developed with the snowball technique. Some generated a random sample within these samples (Good, 1998; Laxton, 2002: Wilke, 1982). The data collection processes for these student projects again are conventional. The number of rounds is usually 3, while up to 5 were required (Kincaid, 2003) due to the increased difficulty of getting consensus from a heterogeneous sample. Not all large, heterogeneous samples required numerous rounds: Lecklitner (1984) used a sample of 345 consisting of 6 subgroups and required only 2 rounds. He did not strive for consensus rather to understand what the subgroups thought about his research questions. While some of the older studies used conventional mail (Cramer, 1990; Lecklitner, 1984; Silverman, 1981), most used electronic mail. However, new technologies (e.g. Questionmark Perception) allow the researcher to put the Delphi questionnaire online where research participants enter their answers. Such answers are in a digital format and then more easily manipulated by the researcher. Some researchers (Cabaniss, 2001; Richards, 2000; V. Schmidt, 1995) used online surveys to collect their data. Finally, the data analysis in these projects varied. Few researchers used purely qualitative analysis,(Kincaid, 2003; Watson, 1982) others quantitative,(Friend, 2001; Krebsbach, 1998; Shook, 1994; Silverman, 1981;Whittinghill, 2000) while most began with qualitative followed by quantitative analysis of subsequent round Likert-style questions (Friend, 2001; Good, 1998; Prestamo, 2000; Richards, 2000; Rosenbaum, 1985). Thus, these graduate student research projects mirror the flexibility seen in other Delphi projects. Delphi Method Design Considerations While the Delphi method is flexible and superficially simple, the researcher needs to take into account many design considerations in order to successfully use the method. Poorly applied like any other research method, the Delphi can yield suspect results. Having used and modified the Delphi method in many research projects in our program, we present some of our insights into Delphi method considerations. Methodological Choices While the Delphi is typically used as a quantitative technique (Rowe & Wright, 1999), a researcher can use qualitative techniques with the Delphi method. Qualitative research is interpretivist in the sense that the researcher is interested in how the social world is interpreted, understood and experienced; the researcher is flexible and sensitive to the social context within which the data was collected; and qualitative research is about producing holistic understandings of rich, contextual and detailed data (Mason, 1996). Qualitative research is also about engaging in conversations with the research participants in a natural setting as opposed to research conducted in a laboratory (Creswell, 1994). The qualitative researcher attempts to make sense of or interpret the phenomena in terms of the meaning the participants place on them (Creswell, 1998). The Delphi method is well suited to rigorously capture qualitative data. It may be seen as a structured process within which one uses qualitative, quantitative or mixed research methods. Such flexibility not only affords the ability of the method to answer many research questions, but also can be well matched to the abilities and aptitudes of the graduate student. The Delphi Method for Graduate Research 10 Initial Question – Broad or Narrow There is a continuum representing the degree of focus or openness of the questionnaire questions. For example, the initial questions are typically broad, open-ended questions so as to widely cast the research net (Adler & Ziglio, 1996; Delbeq et al., 1975; Linstone & Turloff, 1975). Alternatively, the questions can be more focused and structured to guide the Delphi participants towards a certain goal, all the while winnowing down the questions in subsequent rounds. By widely casting the research net in the initial round, one is more likely to get a broader range of responses than if a narrow set of questions were to focus the collective intelligence of the research participants. The tradeoff, however, is that more data is likely to be collected with broad, open-ended questions requiring more time consuming analysis. Focused or broad questions, is a significant decision that needs to be made early in the research design phase. Expertise Criteria The Delphi participants should meet four “expertise” requirements: i) knowledge and experience with the issues under investigation; ii) capacity and willingness to participate; iii) sufficient time to participate in the Delphi; and, iv) effective communication skills (Adler & Ziglio, 1996). Commitment to participate in a multi-round Delphi can be inferred by the round-by-round response rate (Keil et al., 2002). It is our experience that those true experts in a field have great insight; unfortunately, they are often very busy and may not be able to participate fully. Engaging, concise, and well-written questions can often entice their participation. Those with marketing skills often excel at sample development and a high response rate. Often the student's supervisor is a valuable resource to colleagues who qualify as experts. Number of Participants A practical consideration facing the researcher is the sample size. While there are no hard and fast rules, a number of factors should be considered: • Heterogeneous or homogeneous sample: where the group is homogeneous, then a smaller sample of between ten to fifteen people may yield sufficient results. However, if disparate groups are involved (e.g. an international study), then a larger sample will likely be required and several hundred people might participate(Delbeq et al., 1975). A word of caution needs to be extended to the new researcher: heterogeneous groups can greatly increase the complexity and difficulty of collecting data, reaching consensus, conducting analysis, and verifying results. • Decision quality/Delphi manageability tradeoff: there is a reduction in group error (or an increase in decision quality) as sample size increases. However, above a certain threshold, managing the Delphi process and analyzing the data becomes cumbersome in return for marginal benefits. • Internal or external verification: the larger the group, the more convincingly the results can be said to be verified. However, a smaller sample might be used, with results verification conducted with follow-up research. For master theses, often a single Delphi study will often suffice; however, for a PhD dissertation, the Delphi is usually verified with a follow up study (e.g. interviews or survey). There is a wide range in the sample size in these Delphi studies (Table 1 and Appendix). Only three Delphi participants formed the homogeneous sample to develop rules for ceramic casting process, presumably because such expertise is limited (Lam et al., 2000). Conversely, 45 participants were involved from three countries to identify software development risks (R. Schmidt et Skulmoski, Hartman, & Krahn 11 al., 2001). Potential sample size is positively related to the number of experts. One also needs to be cognizant that the views of the sample participants may not be representative of a wider population (Brancheau et al., 1996) which impinges upon results generalization. Cautious interpretation of results is recommended if the sample is small (Nambisan et al., 1999; Wynekoop & Walz, 2000) and/or if the participants' expertise is suspect (Wynekoop & Walz, 2000). Number of Rounds The number of rounds again is variable and dependent upon the purpose of the research. Delbecq, Van de Ven and Gustafson (1975) suggest that a two or three iteration Delphi is sufficient for most research. If group consensus is desirable and the sample is heterogeneous, then three or more rounds may be required. However, if the goal is to understand nuances (a goal in qualitative research) and the sample is homogeneous, than fewer than three rounds may be sufficient to reach consensus, theoretical saturation, or uncover sufficient information. Finally, as the number of rounds increases and the effort required by Delphi participants, one often sees a fall in the response rate (Alexander, 2004; Rosenbaum, 1985; Thomson, 1985). Mode of Interaction There are different modes of Delphi interaction available to the researcher. Initially, the Delphi surveys were pen and paper-based, and often returned through the mail to the researcher (Cramer, 1990; Lecklitner, 1984; Silverman, 1981). This is still an option to the researcher. However, with the advent of electronic mail and personal networked computers, pen and paper-based Delphi’s are less common. Electronic mail affords many advantages to both researcher and Delphi participant alike. Increasingly, experts have access to electronic mail. Perhaps the most significant benefit of electronic mail is the expediency provided by this mode of interaction. Quick turnaround times help to keep enthusiasm alive and participation high. Another benefit of electronic mail is that the raw data is already in a digital format which eliminates the tedious task of transcription. Hartman harnessed group networking technology to complete a one-round Delphi (Hartman & Baldwin, 1995). Finally, the internet allows new ways of group interaction which can be incorporated into the Delphi process (Keil et al., 2002). Others used online surveys (Cabaniss, 2001; Richards, 2000; V. Schmidt, 1995). Thus, there are many different modes of interaction available to the Delphi researcher. Methodological Rigor As with any research, methodological rigor is a cornerstone of “good” research: sloppy research produces sloppy results. Rigor is critical to both quantitative (Creswell, 1994; Fowler, 1993) and qualitative research (Sadleowski, 1986). Rigor is improved when the researcher leaves an audit trail (Sadleowski, 1986). This is a clear decision trail of all key theoretical, methodological and analytical decisions made in the research from beginning to end (Koch, 1994). Audit trails help to substantiate trustworthiness of the research (Rodgers & Cowles 1993). We recommend the researcher regularly use a journal that is dedicated to capturing this information. Thus, methodological rigor can contribute to a successful Delphi – qualitative or quantitative. Results The method of data analysis and results reporting are directly related to the type of questions used in the Delphi instrument. Therefore, researchers need to apply appropriate analysis techniques. The presentation of Delphi results has been more fully discussed elsewhere (Dalkey & Helmer, 1963; Delbeq et al., 1975; Dietz, 1987; Linstone & Turloff, 1975; R. Schmidt, 1997). Some re- The Delphi Method for Graduate Research 12 searchers include analysis of the results sorted according to areas of agreement and disagreement (Keil et al., 2002). Further Verification Many researchers cite as a limitation the difficulty generalizing the results to a wider population due to sample size (Hartman & Jugdev, 1998; R. Schmidt et al., 2001), their limited views or specific agenda (Nambisan et al., 1999; Niederman et al., 1991), and their geographic location (Brancheau et al., 1996). Most researchers recommended further study to refine and verify their results (Keil et al., 2002; Nambisan et al., 1999; Wynekoop & Walz, 2000), to investigate related sets of research questions (Brancheau et al., 1996; Niederman et al., 1991), to extend the results to a similar sample, but from other geographical locations (Brungs & Jamieson, 2005) or to an entirely different sample (R. Schmidt et al., 2001). Verification studies can provide rich research opportunities for new researchers. Publication While some researchers include their Delphi instrument with their publication (Brancheau et al., 1996; Niederman et al., 1991), most do not. In order to tell good numbers from bad numbers, we need to understand not only what was learned, but also how the researchers collected their data. We need to see the instrument and key data (Glass, 1997; Sawyer, 1997). Conclusion The Delphi method is a flexible research technique well suited when there is incomplete knowledge about phenomena. There are many rich research opportunities in the IS discipline that focus on problems, opportunities, solutions and forecasts. The Delphi method would be a suitable candidate for such research projects. It is not just a quantitative method, but works very well in qualitative research. We believe that this method is well suited to IS research because it is a fluid discipline ripe for research. Like IS projects, no two Delphi studies are the same. There are many varieties of Delphi ranging from qualitative to quantitative, to mixed-method Delphi. While there are many varieties of Delphi, common to all are design considerations that need to decided upon including sample composition, sample size, methodological orientation (qualitative and/or quantitative), the number of rounds, and mode of interaction. Considering these choices help to add rigor to the method. Increased rigor contributes to a successful Delphi and deeper understanding of the IS discipline. A final two points. First, the Delphi approach can be aggressively and creatively adapted to a particular situation. Second, when adapting the approach, there is a need to balance validity with innovation. In other words, the greater the departure from classical Delphi, the more likely it is that the researcher will want to validate the results, by triangulation, with another research approach

0 0
Add a comment Improve this question Transcribed image text
Answer #1

Reaserch problem- Investigating implimentation of knowledge management practices in University A and University B.


  1. To identify current knowledge management practices adopted by University A
  2. To identify current knowledge management practices adopted by University B
  3. To compare knowledge management practices of University A and UNiversity B

Sample-: Under this study ,our sampling unit (who will participate in our research) will be Professors, associate professors and assistant professors, IT, administrative, research staff and PHD students of these institutions because as per our opinion educational personnel , IT staff, administration staff and Research students are mainly involved in KM processes, teaching staff and research students are responsible for knowledge acquisition practice, through research and also for knowledge dissemination practice through lectures and counselling processes.. IT staff and administrative staff also involve in KM process and other ICT related activities because they involve in knowledge storage and dissemination through activity of record maintenance and decision making related to organizational practices management. PHD students also involved in this study because they are involved in knowledge acquisition through research and also act as knowledge users.

Round 1 Questionnaire-

Knowledge acquisition practices

  1. My university encourage the performance and cooperation of its employees with other universities on joint research projects.
  2. My university support its students to participate in its research activities.

Knowledge Storage Practice

  1. My university has designated system to store knowledge about the content and implementation of the educational practices.
  2. My university has system to store knowledge about documentation of employees , their abilities and achievements.

Knowledge transfer practices

  1. My university has well designed and efficient system of mentoring the students and young academics.
  2. My university has well designed system that supports the collaboration between its employees.

Knowledge application practices

  1. My university has efficient system and strategies to apply its best practices in the educational processes.
  2. My University has policy for the application of its disposable knowledge in its research and educational potential.

Conclusion/discussion- we are going to use delphi technique for this study.for the

Validation of instruments: Pilot Study will be conducted with a small sample size from the employees of some university to improve the comprehensibility of questionnaire. Reliability of instrument will be checked by using the triangulation of methods such as Expert opinion method and other quantitative techniques for validation of instrument such as Cronbach’s Alpha, and validity by all Items to item correlation. in this study we want to the level of KM practices in both the universities and also compare the result which university has adapted which KM practices the most for eg ; we will compare if University A has adopted Knowledge aquicition practice in all its departments or University B. same way we will compare all other knowledge management practices and come to the conclusion that which one of the universities has focused more on what type of knowledge practices.

Add a comment
Know the answer?
Add Answer to:
First, read the article on "The Delphi Method for Graduate Research." ------ Article is posted below...
Your Answer:

Post as a guest

Your Name:

What's your source?

Earn Coins

Coins can be redeemed for fabulous gifts.

Not the answer you're looking for? Ask your own homework help question. Our experts will answer your question WITHIN MINUTES for Free.
Similar Homework Help Questions
  • Summarize below article 300 words no plagiarism. Thanks  INTRODUCTION Informed consent is a decision to participate...

    Summarize below article 300 words no plagiarism. Thanks  INTRODUCTION Informed consent is a decision to participate in research, taken by a competent individual who has received the necessary information; who has adequately understood the information; and who, after considering the information, has arrived at a decision without having been subjected to coercion, undue influence or inducement, or intimidation. Informed consent is a prerequisite for enrolling human subjects in biomedical research. The concept of “Informed consent” was enshrined in the Nuremberg...

  • I need help with research critique summary of this below article in APA format and in...

    I need help with research critique summary of this below article in APA format and in text citation and the reference en/poni%20perception%20article.pdf EATING DISORDERS 2018, VOL. 26, NO. 2, 107-126,2017.1318624 Routledge Taylor & Francis Group PREVENTION SERIES Check to Perceptions of disordered eating and associated help seeking in young women Annamaria J. McAndrew and Rosanne Menna Department of Psychology, University of Windsor, Windsor, Ontario, Canada ABSTRACT Disordered eating is common among young women, but rates of help-seeking are remarkably...

  • One of the major learning objectives of this chapter is to define a factorial research design,...

    One of the major learning objectives of this chapter is to define a factorial research design, including the terms factor and level, and to identify and describe factorial designs when they appear in a research report. The goal of experiments is to establish the existence of a cause-and-effect relationship between two variables. To accomplish this goal, the researcher manipulates one variable (that is, the independent variable) and measures a second variable. Nonexperiments and quasi-experiments are similar to experiments except that...

  • Directions What is a research proposal? The purpose of a research proposal is to identify and...

    Directions What is a research proposal? The purpose of a research proposal is to identify and formulate questions that arise from nursing practice and to design a research project to study the identified problem. The goal of a research proposal is to present and justify a research idea you have and to present the practical ways in which you think this research should be conducted. Research proposals contain extensive literature reviews and must provide persuasive evidence that there is a...

  • select the correct answer 1) Which statement about systematic reviews is true? a) All systematic reviews...

    select the correct answer 1) Which statement about systematic reviews is true? a) All systematic reviews of quantitative studies are meta-analyses. b) “Systematic review” is just another name for a literature review. c) All systematic reviews of quantitative studies are metasyntheses. d) Systematic reviews follow similar “rules” as for primary studies. 2) Which is the unit of analysis in a meta-analysis? a) Prior systematic reviews b) A primary qualitative study c) Individual study participants d) A primary quantitative study 3)...

  • Use the this tool to critique the article Screening Questions Can't tell No 1. Was there...

    Use the this tool to critique the article Screening Questions Can't tell No 1. Was there a clear statement of the aims of the research? Whet was "he geal of the 'esearch? Why it was thought inportant? hs relevanoe · 。YesOcan't tell。No 2. Is a qualitative methodology appropriate? HINT: Consider If the research seeks to interpret or iluminate the actions and/or subjective expariences of neseanch Is qualitative research the right methodolagy for addressing the research ? Is it worth continuing?...

  • Case Study Quiz of Research Method

    x.øi5QuestionsRead the above Case study and give the answers of the questions byusing correct options.1. What is the primary focus of establishment surveys in this case study?A. Collect the data through past studiesB. Analyze the literature reviewC. Using of quantitative techniquesD. Data collection through mail and Interview2. Which one of following is generally common in establishment survey and householdsurvey?A. Cognitive recallB. Homogenous respondentsC. Error freeD. Response burden3. Which one of the following is not of important consideration in establishment surveywhile...

  • What is the qualitative method? to this article 2. What type of sampling is used? Is...

    What is the qualitative method? to this article 2. What type of sampling is used? Is it appropriate for this method? Abstract This qualitative study used interviews to explore nurses’ perceptions of their role in protecting children and to identify any barriers to implementing the role. Participants in this study were recruited through purposive sampling. Ten nurses whose work brings them into regular contact with children and their families were interviewed; the sample included nurses from acute care and community-based...

  • Answer the questions based on following information. 7.     Describe the data collection method(s)'...

    Answer the questions based on following information. 7.     Describe the data collection method(s)' a.     Who collected the data? b.     What tools were used? c.     What were the ethical considerations addressed and discuss gaps you identified. d.     You will need to summarize and analyze the information from the article in your own words Method a quantitative, cross-sectional, descriptive design was chosen for this study. The instrument that was used to collect the data was a self-report questionnaire. a questionnaire previously used...

  • 1. What was the research question for the meta-analysis? 2. Why was the Maslach Burnout Inventory...

    1. What was the research question for the meta-analysis? 2. Why was the Maslach Burnout Inventory (MBI) included in the selection criteria? 3. What was the final sample size? 4. What were the results of the meta-analysis? 5. What is the personal critique of the meta-analysis? PLEASE TYPE :) thank you PsycINFO, SciELO, and Scopus. ProQuest Dissertation&included the items specifically related to the internal Thesis and Google Scholar were used to find gray litera lidity of the study. We used...

Free Homework Help App
Download From Google Play
Scan Your Homework
to Get Instant Free Answers
Need Online Homework Help?
Ask a Question
Get Answers For Free
Most questions answered within 3 hours.