Chapter Four: Assessment Methods

It is now time to determine how the outcomes/objectives will be measured. When you are determining the success of student learning or of a performance objective, you need tools or assessment methods to measure with. It might be easier to think in terms of what you are trying to measure:

• Are you trying to measure an employer’s satisfaction with the quality of the program graduate?
• Are you trying to measure the results of a recruitment campaign for minority students?
• Are you trying to measure the student’s ability (behavioral) to perform a specific skill?
• Are you trying to measure the student’s perception (affective) of the value of the internship completed through the program?
• Are you trying to measure the student’s knowledge (cognitive) of learning that resulted from completion of the program?
• Are you trying to measure the unit’s efficiency in processing applications in a timely manner?
• Are you trying to measure the alumni’s interest in supporting the University?

Direct and Indirect Assessment

Direct measures evaluate actual performance such as reflected in a report about faculty scholarly activity or unit productivity, or student competencies as reflected in a test score or in a capstone course. Indirect measures evaluate perceived performance such as the satisfaction with timeliness of services rendered, approval of the teaching style of the instructor, or the employer’s perception of the graduate’s skill.

Grades

A grade is earned by an individual student and assigned by a particular instructor in relation to a specific course. A group of courses in targeted combination formulates an entire program. In conducting assessment, an individual grade is a focused assessment of a particular student’s learning. Course assessment has merit for improving delivery and content in an effort to improve student learning. To assess a program, you take a broader approach by assessing the learning outcomes of the program’s graduates in its entirety. Program assessment is intended to improve student learning outcomes.

Programs relying on grades alone have inherent flaws in their assessment. Instructors grade on different scales using varying criteria. Course material and delivery may vary by instructor and may change semester to semester. Two students taking the same course, during the same semester, receiving the same amount of credit, but enrolled in different sections may complete

the course with very different learning outcomes. For these reasons, grades alone are not valid indicators of a program’s student learning outcomes. Grades can be used in combination with other assessment methods, as long as it is clearly understood and specified how the grade is assigned. The best assessment plan will include multiple measures taken at varying points throughout the program.

Surveys

A survey is a series of questions designed to obtain feedback from a targeted audience. The most important factor in developing a survey is making sure you have defined the research question. Surveys are easy to administer and provide immediate feedback. However, surveys can also be difficult to design effectively and they don’t always contain constructive responses. The return response rate for surveys can be very low, diminishing their effectiveness. Examples include alumni surveys, employer surveys, service satisfaction surveys, student life surveys, student opinion surveys, and graduating student surveys. The survey objectives should state who, what, where, when, and why. An example research question is:

What? What is the level of satisfaction with the services provided by
Who? the Office of Institutional Effectiveness
Where? at McNeese State University
When? in the 2005-06 calendar year
Why? to establish base line data to monitor progress toward improving institutional assessment.

Focus Groups

A focus group is a planned discussion with a targeted audience for purposes of soliciting input. A typical focus group includes seven to ten participants who are chosen because they have a common thread that ties them to the subject being discussed. The moderator’s role should be as listener. It is best to use an impartial moderator not connected to the group. Focus groups may occur in any venue and with any group of individuals. As such, the environment for the focus group is generally casual and the atmosphere conducive to discussion. A good moderator can often introduce an icebreaker to relax attendees and to encourage feedback. However, it may be difficult to analyze the discussion that results from the focus group. A focus group may not always contain a representative group of targeted members. Attendees may be reluctant to openly express themselves in a group setting. Examples of a focus group include a student focus group, an employer focus group, a graduate focus group, an employee focus group, and an exit interview focus group.

Exams

Exams may be used to assess knowledge, skills, and abilities. Exams may be ready-made or “canned” purchases or they can be developed internally. Purchased exams may provide greater validity and reliability than an internally developed exam. However, developing an exam internally gives the instructor more latitude to embed questions for assessment of student learning. Questions consistently answered incorrectly can be changed or teaching methods modified to improve learning outcomes. However, instructors vary in the type of exams given and in the quality of exams prepared. Exams developed in-house are not always valid or reliable. When relying on exams to assess student learning, an instructor must be cognizant of avoiding teaching for test purposes rather than teaching for student learning. Examples include instructor- designed oral exams, nationally-normed written exams, and pre- and post- exams.

Student Academic Performance

Students are evaluated at many stages of matriculation using different assessment methods. The method of evaluation employed often depends on the field of study. For example, an art major may work on a student portfolio throughout the program for presentation during the final semester. Measuring a student’s academic performance provides an excellent indicator of the level of competency achieved through completion of the program. Assessing student performance over time provides a higher level of validity than assessing performance at a specific point in time. However, if rubrics have not been established, it may be difficult to evaluate objectively. Sample sizes may be limited based on the number completing the program. Examples include student portfolios, capstone projects, thesis/dissertation defense, oral presentations, and research presentations.

Internal Data

Internal data is current or archived data that relates to the institution, the units, or the programs. Generally, ample data has already been collected for use in internal assessment. Utilizing existing data reduces the cost of collecting new data. Data collected over time can show patterns and trends. However, users may lack knowledge regarding the data that is available and/or lack understanding of the meaning behind the data. It can be costly to collect new data, particularly if a canned data collection method is used. Examples include grade distribution over time, retention data trends, enrollment data trends, graduation trends, and student demographic trends.

Rubrics

A rubric, an instrument used in an effort to objectively evaluate a student’s performance, forms the basis of assessment for student work. Assessing student learning based on grades alone is not enough. Instead, grades may be tied to a scoring rubric to identify learning areas for improvement. Rubrics are effective because they provide objectivity in grading and consistency in evaluating student work. Rubrics set a standard of expectations for student work as well as help students evaluate their own work. To construct a rubric:

• Determine the essential components that you are using to evaluate student work. What are you basing your grading system on?
• Determine the levels of competency that you want to use as measures in the grading assessment. Four levels are the most commonly used and are easier to distinguish when rating. An example might be 1) Exemplary, Proficient, Marginal, Unacceptable; 2) Strong, Acceptable, Weak, Not Acceptable; 3) Advanced, Mastery, Aware, Unaware; or 4) Excellent, Good, Fair, Poor.
• For each level of competence, provide a clear, detailed description of what you are looking for in the expected performance. Use objective descriptors, not value statements and include both qualitative and quantitative differences.
• Test your rubric to see if it holds up. Try to identify loopholes such as late assignments, wrong formatting, or plagiarism so they can be corrected prior to implementing the rubric. Make revisions when needed.

Remember:

• Effective assessment involves more than one method of measurement.
• Use both direct/indirect and qualitative/quantitative measures.
• If you consider yourself a novice in assessment, start out small. It is more important to measure two to three outcomes/objectives well than trying to measure every outcome/objective you may have.
• The assessment method should be selected on the basis of the outcome/objective specified.
• Use data and information that is already being compiled, if available. You can always add more data when appropriate.

Useful Information Regarding Assessment Methods

Academic/Administrative Support Rubrics at Rollins College
http://www.rollins/edu/provost/asssessment
Using Course Grades for Assessment
http://www.ncsu.edu/provost/academic_programs/uapr/FAQ/UAPRFAQwhynotcoursegrades.html
Portfolio Project Builder
http://catalyst.washington.edu/catalyst/how-to/portfolio/index.html
Sample Syllabus and Rubrics
http://www.educ.iastate.edu/elps/resev/RE561.HgEd597.htm
Course Mapping
http://www.bridgew.edu/AssessmentGuidebook/chapter4.cfm#focusing
Rubrics
http://www.bridgew.edu/AssessmentGuidebook/chapter6.cfm#rubrics
Assessment Frequently Asked Questions (FAQs)
http://www.utsystem.edu/aca/assessment/faq.htm