Question

Discuss the pros and cons of assessing performance in quality measurement activities.Discuss the pros and cons of assessing performance in quality measurement activities.

Answer 1

Ans)As defined by the IOM study committee, quality of care is the degree to which health services for individuals and populations increase the likelihood of desired health outcomes and are consistent with current professional knowledge.

- Measures are based on scientific evidence about processes, outcomes, perceptions, or systems that relate to high quality care5. Performance measures, simply put, allow practitioners to measure clinical care delivered, because one cannot change or improve things that one cannot measure.

- Information on a variety of instruments useful for doing assessment is given below.
1. Tests
a. Commercial, norm-referenced, standard examinations
b. Locally developed written examinations (objective or subjective designed by faculty);
c. Oral examinations (evaluation of student knowledge levels through a face-to-face interrogative
dialogue with program faculty).
2. Competency-Based Methods
a. Performance Appraisals - systematic measurement of overt demonstration of acquired skills
b. Simulations
c. “Stone” courses (primarily used to approximate the results of performance appraisal, when direct
demonstration of the student skill is impractical).
3. Measures of Attitudes and Perceptions (can be self-reported or third party)
a. Written surveys and questionnaires (asking individuals to share their perceptions of their own or
others' attitudes and behaviors including direct or mailed, signed or anonymous).
b. Exit and other interviews (evaluating reports of subjects' attitudes and behaviors in a face-to-face
interrogative dialogue).
c. Focus groups
4. External Examiner (using an expert in the field from outside your program – usually from a similar program
at another institution – to conduct, evaluate, or supplement the assessment of your students).
5. Behavioral Observations – including scoring rubrics and verbal protocol analysis (measuring the frequency,
duration and topology of student actions, usually in a natural setting with non-interactive methods).
6. Archival Records (biographical, academic, or other file data available from college or other agencies and
institutions).
7. Portfolios (collections of multiple work samples, usually compiled over time).

Norm-Referenced, Standardized Exams
Definition: Group administered, mostly or entirely multiple-choice, “objective” tests in one or more curricular areas.
Scores are based on comparison with a reference or norm group. Typically must be obtained (purchased) from a private vender.
Target of Method: Used primarily on students in individual programs, courses or for a particular student cohort.
Advantages:
• Can be adopted and implemented quickly
• Reduce/eliminate faculty time demands in instrument development and grading (i.e., relatively low “frontloading” and “backloading” effort)
• Objective scoring
• Provide for externality of measurement (i.e., external validity is the degree to which the conclusions in your study would hold for other persons in other places and at other times – ability to generalize the results
beyond the original test group.)
• Provide norm reference group(s) comparison often required by mandates.
• May be beneficial or required in instances where state or national standards exist for the discipline or profession.
• Very valuable for benchmarking and cross-institutional comparison studies.
Disadvantages:
• May limit what can be measured.
• Eliminates the process of learning and clarification of goals and objectives typically associated with local development of measurement instruments.
• Unlikely to completely measure or assess the specific goals and objectives of a program, department, or institution.
• “Relative standing” results tend to be less meaningful than criterion-referenced results for program/student evaluation purposes.
• Norm-referenced data is dependent on the institutions in comparison group(s) and methods of selecting students to be tested. (Caution: unlike many norm-referenced tests such as those measuring intelligence,
present norm-referenced tests in higher education do not utilize, for the most part, randomly selected or well stratified national samples.)
• Group administered multiple-choice tests always include a potentially high degree of error, largely uncorrectable by “guessing correction” formulae (which lowers validity).
• Summative data only (no formative evaluation)
• Results unlikely to have direct implications for program improvement or individual student progress
• Results highly susceptible to misinterpretation/misuse both within and outside the institution
• Someone must pay for obtaining these examinations; either the student or program.
• If used repeatedly, there is a concern that faculty may teach to the exam as is done with certain AP high.

Locally Developed Exams
Definition: Objective and/or subjective tests designed by faculty of the program or course sequence being
evaluated.
Target of Method: Used primarily on students in individual classes, a specific program of interest, or for a
particular cohort of students
Advantages:
• Content and style can be geared to specific goals, objectives, and student characteristics of the program,
curriculum, etc.
• Specific criteria for performance can be established in relationship to curriculum
• Process of development can lead to clarification/crystallization of what is important in the process/content
of student learning.
• Local grading by faculty can provide relatively rapid feedback.
• Greater faculty/institutional control over interpretation and use of results.
• More direct implication of results for program improvements.
Disadvantages:
• Require considerable leadership/coordination, especially during the various phases of development
• Cannot be used for benchmarking, or cross-institutional comparisons.
• Costly in terms of time and effort (more “frontloaded” effort for objective; more “backloaded” effort for subjective)
• Demands expertise in measurement to assure validity/reliability/utility
• May not provide for externality (degree of objectivity associated with review, comparisons, etc. external to the program or institution).

Oral Examination
Definition: An evaluation of student knowledge levels through a face-to-face interrogative dialogue with program
faculty.
Target of Method: Used primarily on students in individual classes or for a particular cohort of students
Advantages
• Content and style can be geared to specific goals, objectives, and student characteristics of the institution,
program, curriculum, etc.
• Specific criteria for performance can be established in relationship to curriculum
• Process of development can lead to clarification/crystallization of what is important in the process/content
of student learning.
• Local grading by faculty can provide immediate feedback related to material considered meaningful.
• Greater faculty/institutional control over interpretation and use of results.
• More direct implication of results for program improvements.
• Allows measurement of student achievement in considerably greater depth and breadth through follow-up
questions, probes, encouragement of detailed clarifications, etc. (= increased internal validity and formative
evaluation of student abilities)
• Non-verbal (paralinguistic and visual) cues aid interpretation of student responses.
• Dialogue format decreases miscommunications and misunderstandings, in both questions and answers.
• Rapport-gaining techniques can reduce “test anxiety,” helps focus and maintain maximum student attention
and effort.
• Dramatically increases “formative evaluation” of student learning; i.e., clues as to how and why they
reached their answers.
• Identifies and decreases error variance due to guessing.
• Provides process evaluation of student thinking and speaking skills, along with knowledge content.
Disadvantages
• Requires considerable leadership/coordination, especially during the various phases of development
• Costly in terms of time and effort (more “frontload” effort for objective; more “backload” effort for subjective)
• Demands expertise in measurement to assure validity/reliability/utility
• May not provide for externality (degree of objectivity associated with review, comparisons, etc. external to the program or institution).
• Requires considerably more faculty time, since oral exams must be conducted one-to-one, or with very small groups of students at most.
• Can be inhibiting on student responsiveness due to intimidation, face-to-face pressures, oral (versus written) mode, etc. (May have similar effects on some faculty!)
• Inconsistencies of administration and probing across students reduces standardization and generalizability of results (= potentially lower external validity).

Performance Appraisals
Definition: A competency-based method whereby pre-operationalized abilities are measured in most direct, real-
world approach. Systematic measurement of overt demonstration of acquired skills.
Target of Method: Used primarily on students in individual classes or for a particular cohort of students
Advantages:
• Provide a more direct measure of what has been learned (presumably in the program)
• Go beyond paper-and-pencil tests and most other assessment methods in measuring skills
• Preferable to most other methods in measuring the application and generalization of learning to specific settings, situations, etc.
• Particularly relevant to the goals and objectives of professional training programs and disciplines with well defined skill development.
Disadvantages:
• Ratings/grading typically more subjective than standardized tests
• Requires considerable time and effort (especially front-loading), thus being costly
• Sample of behavior observed or performance appraised may not be typical, especially because of the presence of observers.

Simulations
Definition: A competency based measure whereby pre-operationalized abilities are measured in most direct, real-
world approach. Simulation is primarily utilized to approximate the results of performance appraisal, but when – due to the target competency involved, logistical problems, or cost – direct demonstration of the student skill is impractical.
Target of Method: Used primarily on students in individual classes or a group of students
Advantages
• Better means of evaluating depth and breadth of student skill development than tests or other performance-based measures (= internal validity).
• More flexible; some degree of simulation can be arranged for virtually any student target skill.
• For many skills, can be group administered, thus providing and excellent combination of quality and economy.
Disadvantages
• For difficult skills, the higher the quality of simulation the greater the likelihood of the problems of performance appraisal; e.g., cost, subjectivity, etc. (see “Performance Appraisals”).
• Usually requires considerable “frontloading” effort; i.e., planning and preparation.
• More expensive than traditional testing options in the short run.

“Stone” Courses
- Often not considered an assessment method in itself.
Definition: Courses, usually required for degree/program completion, which in addition to a full complement of instructional objectives, also serve as primary vehicles of student assessment for program evaluation purposes;
e.g., Capstone, Cornerstone, and Keystone courses.
Advantages:
• Provides for a synergistic combination of instructional and assessment objectives.
• A perfect mechanism for course-embedded assessment of student learning and development (i.e., outcomes, pre-program competencies and/or characteristics, “critical indicators,” etc.)
• Can add impetus for design of courses to improve program orientation/integration/updating information for students.
Disadvantages:
• None specified

Behavioral Observations
Definition: Measuring the frequency, duration, topology, etc. of student actions, usually in a natural setting with
non-interactive methods. For example, formal or informal observations of a classroom. Observations are most
often made by an individual and can be augmented by audio or videotape.
Target of Method: Used primarily on individuals or groups of students in classes
Advantages
• Best way to evaluate degree to which attitudes, values, etc. are really put into action (= most internal
validity).
• Catching students being themselves is the most “natural” form of assessment (= best external validity).
• Least intrusive assessment option, since purpose is to avoid any interference with typical student activities.
Disadvantages
• Always some risk of confounded results due to “observer effect;” i.e., subjects may behave atypically if they
know they’re being observed.
• Depending on the target behavior, there may be socially or professionally sensitive issues to be dealt with
(e.g., invasion of privacy on student political activities or living arrangements) or even legal considerations
(e.g., substance abuse or campus crime).
• May encourage “Big Brother” perception of assessment and/or institution.
• Inexperienced or inefficient observers can produce unreliable, invalid results.

Portfolios
Definition: Collections of multiple student work samples usually compiled over time. Rated by some type of rubric.
Target of Method: Used primarily on students in individual classes or in for a particular cohort of students
Advantages:
• Can be used to view learning and development longitudinally (e.g. samples of student writing over time can
be collected), which is most valid and useful perspective.
• Multiple components of a curriculum can be measured (e.g., writing, critical thinking, research skills) at the
same time.
• Samples in a portfolio are more likely than test results to reflect student ability when pre-planning, input
from others, and similar opportunities common to most work settings are available (which increases
generalizability/external validity of results).
• The process of reviewing and grading portfolios provides an excellent opportunity for faculty exchange and
development, discussion of curriculum goals and objectives, review of grading criteria, and program
feedback.
• Economical in terms of student time and effort, since no separate “assessment administration” time is required.
• Greater faculty control over interpretation and use of results.
• Results are more likely to be meaningful at all levels (i.e., the individual student, program, or institution) and can be used for diagnostic/prescriptive purposes as well.
• Avoids or minimizes “test anxiety” and other “one shot” measurement problems.
• Increases “power” of maximum performance measures over more artificial or restrictive “speed” measures on test or in-class sample.
• Increases student participation (e.g., selection, revision, evaluation) in the assessment process.
Disadvantages
• Costly in terms of evaluator time and effort.
• Management of the collection and grading process, including the establishment of reliable and valid grading criteria, is likely to be challenging.
• May not provide for externality.
• If samples to be included have been previously submitted for course grades, faculty may be concerned that a hidden agenda of the process is to validate their grading.
• Security concerns may arise as to whether submitted samples are the students’ own work, or adhere to other measurement criteria.