Lessons in measuring ‘learning gain’: One size fits none

Adopting a single sector-wide measure to capture ‘learning gain’ – the university student development journey from fresher to graduate – lacks validity according to research conducted by academics at Sheffield Hallam University.

Researchers in the directorate of Student Engagement, Evaluation and Research, were asked by the Office for Students to carry out an Evaluation of the National Mixed Methods Learning Gain Project (NMMLGP) and Student Perceptions of Learning Gain for the higher education sector and spoke with nine institutions as part of their evaluation research. Those questioned ranged from niche providers to Russell Group universities.

The original aim of the evaluation research was to understand the feasibility of administering a sector-wide learning gain questionnaire.

 

During the first phase (December 2017 and April 2018) the researchers conducted interviews with senior staff from each of the nine institutions. They  found there was dissatisfaction about survey design, including length, complexity and removal from subject/disciplinary focus. Additionally, there were major communication and tracking difficulties concerning students whom had taken the test.

Stella Jones-Devitt, who led the evaluation research project at Sheffield Hallam University, said:

 

“There was a difference between what the sector was trying to do and what the students wanted. There has been a huge amount of investment by the sector to try and work out what the perfect student ‘journey’ looks like. However, we discovered that it wasn’t possible to produce a one size fits all tool to measure the ultimate student experience.”

During phase two of the project (October to December 2018) the original online questionnaire for students was replaced with a more student-centred qualitative approach, using activity-based focus groups led by student researchers.

The outcome was that most participants demonstrated low to moderate awareness of the term ‘learning gain’ with many unaware of the concept. They agreed that a learning gain measure needed to be flexible to meet individual circumstances and should be embedded within the subject being studied if to be of real value.

Overall the research team found that a one-size fits all measure of learning gain held minimal value for students and should not be pursued further; the area needs more exploration and the sector needs to consider whose interests are best served by the measurement of learning gain.

Should it be a marker of institutional positioning within a competitive higher education market or a process of progression through the student journey? Findings from the evaluation research conducted by Sheffield Hallam suggest that this question is yet to be resolved.

The Survey Design Checklist, which was one approach used to evaluate the NMMLGP methodology, can be accessed here: SRDC.

Image courtesy of Pixabay