Analytics

Analytics at OVPIUE

The Office of the Vice-Provost, Innovations in Undergraduate Education (OVPIUE) leads the development of new data tools to help our faculty and tri-campus partners enact change and enhance the academic experience. Whether it’s academic leaders examining trends in student retention or individual course instructors learning more about student patterns of activity in Quercus, OVPIUE is working to support data-driven decision-making and informed action. The two primary projects our office is involved in are Academic Analytics and Quercus Learning Analytics.

Academic Analytics

In 2017, the OVPIUE launched a tri-campus initiative, (formerly called Student Academic Success) with the goal of establishing a robust suite of data tools to support academic divisions and units to better understand the retention, progression, and graduation patterns of their undergraduate students. Each tool in the suite examines student patterns from a specific level of analysis: institutional, divisional, within units, within Subject Programs of Study (SPOSts), or at the course level. Prior to this, a Provostial or Divisional office would curate their own data, and create their own definitions in response to an ad-hoc request for data. The project provides appropriate members of the institution with responsibility for curriculum design, program assessment, and student academic success with a series of dashboards to help them make decisions about program design, enrollment management, and student support initiatives.

The Academic Analytics project is under the stewardship of the VPIUE but was developed in consultation with the first-entry undergraduate divisions[1] and in partnership with the Institutional Research and Data Governance (IRDG) office. These specific tools were developed in response to divisional data priorities and needs related to undergraduate student academic success throughout the degree and subject program lifecycle. To develop each tool in the suite, OVPIUE worked with the academic divisions to identify the right strategic and operational leaders to enact the work, coordinated the strategic and operational working groups, led all aspects of project support, and we continue to oversee training, support and data access.

These partnerships and collaborations have resulted in four data tools: 

  1. Institutional & Divisional Retention, Progression, and Graduation: Retention, progression, and graduation from both the institutional and divisional levels. Includes data from all first-entry undergraduate divisions as of 2007 onward. 
  2. Individual Course Performance: Cancellations, waitlists, withdrawals and failures at the individual course level. Includes data from all first-entry undergraduate divisions as of 2007 onward. 
  3. Subject Program of Study (SPOSt) Admissions: Metrics around application, offers and acceptances into subject programs of study. Includes data as of 2013 for the Faculty of Arts & Science, the University of Toronto Mississauga, and the University of Toronto Scarborough. 
  4. Subject Program of Study (SPOSt) Retention: Retention and graduation based on the initial set of subject programs a student was enrolled in. Includes data as of 2007 for the Faculty of Arts & Science, the Daniels Faculty of Landscape, Architecture and Design, the University of Toronto Mississauga, and the University of Toronto Scarborough. 

With the tools live we continue to act as a champion for student success, supporting divisional onboarding and training of new users, chairing an advisory group to receive feedback on dashboard improvements, and working with divisions to help make meaning of the data and consider initiative to better support student success.  

For more information about Academic Analytics or to request access to the Academic Analytics dashboards, contact jeff.burrow@utoronto.ca. 

Quercus Learning Analytics

At the University of Toronto, learning analytics refers to what we can learn from student activity data within our enterprise learning system, such as patterns of digital activity in their courses, to inform the design of courses, assessments, and academic programs.  

University of Toronto instructors indicated an interest in the potential benefits of improving available analytic tools to provide better insight to learner activity within the Quercus (Canvas Learning Management System) environment. In 2020-21, the Vice-Provost, Innovations in Undergraduate Education sponsored a Tri-campus consultation process to identify and prioritize opportunities and general recommendations for potential applications of learning analytics to better support student academic achievement. This process culminated in the creation of a Strategy Paper that highlighted three key goals for the University:

  1. Improve student learning experiences and outcomes.
  2. Enable instructors and staff to access and leverage learning data in support of teaching and learning activities.
  3. Optimize the structure and support of our digital learning environment.

The overall Learning Analytics project is under the stewardship of the OVPIUE. The VPIUE chairs the steering committee. We have and continue to support, individual projects with one-time funding for faculty-led projects, investments in infrastructure to build the record store, and ongoing project management and coordination.

The major projects resulting from this strategy paper include:

  1. Data-Driven Design (D3: Quercus Analytics):  This initiative provides support to instructors to use student Quercus activity data to make course redesign decisions aimed at improving instruction and the learner experience. A cohort of six instructors explored Quercus learner data through analysis of downloaded course activity and quiz reports with support from learning data analysts.
  2. Instructor-Facing Quercus Dashboard: In planning for a set of user-friendly dashboards, which can be accessed by any instructor to inform course design and pedagogical outcomes, we have actively sought feedback from instructors in a variety of forums to enhance usability.
  3. Program Level Deep Dive: This project is engaging a departmental team of course instructors and program administrators in collaboratively developing a series of questions that explore how course-level learning analytics data can inform program-level design.  
  4. Quercus Record Store: The Quercus Record Store (QRS), will allow the University of Toronto (U of T) to leverage Canvas Data Services (CDS) to ingest, store, process and analyze Quercus usage data securely and at scale.  Data from the QRS will be integrated into the institutional data hub and will power the Instructor-Facing Quercus Dashboards
Visit the Learning Analytics website

[1] Faculty of Arts & Science, University of Toronto, Mississauga, University of Toronto Scarborough, Faculty of Applied Science & Engineering, Faculty of Kinesiology and Physical Education, Daniels Faculty of Landscape & Design, Faculty of Music