Using Institutional-level Rubrics to Assess Student Achievement of Institutional Learning Outcomes
Curcio (2018) suggested that the use of rubrics is an accepted method in assessing learning outcomes in undergraduate education. Extending the use of existing rubrics to institutional learning outcomes (e.g., Simper et al., 2018), SAIL aims to determine the utility of faculty-created institutional rubrics for assessing student achievement of ILOs in courses that have been designated as meeting an ILO.
During the first iteration of SAIL, faculty collaboratively developed rubrics, sought student consent, and assessed anonymized students’ assignments to determine the degree of student achievement of three ILOs: Critical Thinking and Investigation, Social Responsibility, and Lifelong Learning. Piloting three ILOs is adequate for testing as demonstrated by Norman’s (2017) use of three VALUE (AAC&U) rubrics: critical thinking, quantitative literacy, and written communication.
For each ILO, a rubric was developed by faculty in sessions facilitated by an educational developer and quality assurance practitioner. The rubrics included consideration of theoretical principles, knowledge, reflection, application, and other skills that align with assessable knowledge, skills, and attitudes (Stassen et al., 2004).
Co-Creation of Shared Institutional Rubrics
The ILO Rubrics are based on the criteria listed in the ILO Foci Tools used at the university under study for identifying courses that meet institutional learning outcomes.
During the first iteration of SAIL, faculty within each ILO Pod were tasked with determining the number of levels (columns) and associated titles, as well as co-writing the descriptions (cells) of each level of performance (column) for each criteria (row). Additional faculty were involved at this stage of rubric co-creation to broaden disciplinary perspectives, as needed.
Following Pilot #1, feedback during the Debrief was incorporated into the revised rubrics. During the second iteration of SAIL, faculty within each ILO Pod were tasked with reviewing the descriptions and refining the rubrics, as appropriate. So far, through this research, we have collaboratively developed four ILO Rubrics (Table 2.1).
| Institutional Learning Outcome |
ILO Rubric |
|---|---|
| Communication | Rubric – Communication Rubric (PDF) |
| Lifelong Learning | Rubric – Lifelong Learning (PDF) |
| Social Responsibility | Rubric – Social Responsibility (PDF) |
| Critical Thinking and Investigation | Rubric – Critical Thinking and Investigation (PDF) |
The rubric levels (column headings) of Beginning, Developing, Meeting, and Exceeding were chosen by faculty during the first pilot to reflect student growth over time as they progress through their degree. The levels reflect that some students are beginning to demonstrate the ILO, while others are at the meeting level. Across programs and within a degree, a course that is introducing the ILO may be satisfied with achieving the beginning level. For example, a first-year course may expect and see most students at the beginning or approaching levels. Whereas, an upper-year course may expect and see most students at the approaching and meeting levels if still a new outcome for students, or at the meeting and exceeding levels where taught and practiced multiple times during the program.
The rubric categories/criteria (rows) are the ILO foci. The foci were developed to provide evidence of a substantive match between the ILO foci and a course’s learning outcomes. The ILO foci refer to the central topic or intent of an ILO with four areas in mind (analysis, theory, application/demonstration, reflection) that are applicable from first through fourth year of baccalaureate degree programs, and consider the diversity of disciplinary principles and methodologies. Below are the Foci Tools for each ILO (Table 2.2; Thompson Rivers University, 2020).
| Institutional Learning Outcome |
Foci Tool |
|---|---|
| Communication | COMMUNICATION (PDF) |
| Teamwork | TEAMWORK (PDF) |
| Social Responsibility | SOCIAL RESPONSIBILITY (PDF) |
| Lifelong Learning | LIFELONG LEARNING (PDF) |
| Critical Thinking and Investigation | CRITICAL THINKING and INVESTIGATION (PDF) |
| Knowledge | KNOWLEDGE (PDF) |
| Intercultural Awareness | INTERCULTURAL AWARENESS (PDF) |
| Indigenous Knowledges and Ways | INDIGENOUS KNOWLEDGES AND WAYS (PDF) |
Each ILO has approximately five to eight foci that characterize the ILO. A course only needs to meet three of the ILO foci to be deemed a substantive match. The ILO foci were used as the categories (rows) for the shared institutional rubrics.
The descriptors (where each criterion intersects with a level) were created by SAIL faculty members in 2020-21 for Social Responsibility, Critical Thinking and Investigation, and Lifelong Learning. One of the primary tasks of each subsequent pilot and use of the rubric includes a review of the descriptors at the course level, as well as at the institutional level. This ongoing opportunity to provide feedback ensures that the rubrics continue to attend to cross-disciplinary utility.
Rubric Creation Workshop
Rubric Creation Workshops are facilitated by trained faculty and are customized for each ILO Pod. The workshop is roughly two hours in duration. Below are instructions for SAIL Coordinators to facilitate a Rubric Creation Workshop with participating faculty members (Table 2.3).
| Duration | Activity | Resources |
|---|---|---|
| 15 min. | Review ILO Foci Tool and ILO foci. | ILO Foci Tool |
| 20 min. | Determine the number of levels (column headings) and brainstorm titles for each level. | Sample rubrics (e.g., AAC&U Value Rubrics) |
| 20 min. | Brainstorm key words for each of the descriptors. | Sample rubrics (e.g., AAC&U Value Rubrics) |
| 20 min. | Draft descriptors. | |
| 30 min. | Discuss assignments within participating faculty members’ courses that best align with the rubric; discuss which foci are relevant and confirm that the descriptors make sense for the chosen assignment. Revise the descriptors, as appropriate based on the discussion. | |
| 15 min. | Wrap up and discuss next steps. Remind faculty members of the student consent process. | Data Collection Notice (PDF) |
References
Allen, M. J. (2008). Strategies for direct and indirect assessment of student learning. SACS-COC Summer Institute.
Association of American Colleges and Universities. (2009). Valid Assessment of Learning in Undergraduate Education (VALUE). AAC&U. https://www.aacu.org/initiatives/value
Curcio, A. A. (2018). A simple low-cost institutional learning-outcomes assessment process. Journal of Legal Education, 67(2), 489-530.
Kuh, G. D., Ikenberry, S. O., Jankowski, N. A., Cain, T. R., Ewell, P. T., Hutchings, P., & Kinzie, J. (2015). Using evidence of student learning to improve higher education. Jossey-Bass.
National Institute for Learning Outcomes Assessment (2016, May). Higher education quality: Why documenting learning matters. University of Illinois and Indiana University, NILOA.
Norman, C. R. (2017). Students’ performance on institutional learning outcomes. Retrieved from https://repository.stcloudstate.edu/cgi/viewcontent.cgi?article=1013&context=hied_etds
Nunley, C., Bers, T., & Manning, T. (2011). NILOA’s learning outcomes assessment in community colleges. Retrieved from: www.learningoutcomesassessment.org/documents/CommunityCollege.pdf
Simper, N., Frank, B., Scott, J., & Kaupp, J. (2018). Learning outcomes assessment and program improvement at Queen’s University (pp. 1–53). Higher Education Quality Council of Ontario (HECQO).
Stassen, M. L.A., Doherty, K., & Poe, M. (2004). Program-based review and assessment: Tools and techniques for program improvement. Retrieved from www.umass.edu/oapa/sites/default/files/pdf/handbooks/program_assessment_handbook.pdf
Thompson Rivers University. (2020). ILO Foci Tools. Author.
