Evaluating Learning and Teaching

Evaluating your teaching and student learning is an important part of being a lecturer as it allows you to understand how to adapt and improve your practice. Below are some key methods that can be done before, during and after a teaching activity or series of activities to evaluate student engagement, student learning and your teaching.

Before, during and immediately after the teaching activity/series of activities (formative evaluation)

Before the activity/series of activities:

  • Ask students to send through questions or share ideas about any flipped content in advance. This allows for co-designed discussions; you can also check whether any students are having problems accessing the content.
  • Establish a discussion forum/social media chat to create parallel discussion of the pre-session learning materials e.g. announcement to say the pre-session learning is live; encourage social consumption with the group (e.g. ‘You Tube premieres); encourage use of ’likes’ or constructive critique.
  • Use the objectives set for the session/s to guide this evaluation and reflection. Were the objectives for the session/s met? Can you evidence this?

During the activity/series of activities:

  • In blended sessions, encourage students from the in-room session to engage in discussion with remote students. Support students to act as facilitators in the break-out rooms.
  • Embed regular checks to make sure both in-room and remote groups are both at the same stage. (Zoom has a slower/faster button, for example, for participants to feedback on the pace of a session).
  • Make sure there is a live chat/Q&A function and that it is being monitored/responded to. Students can be appointed to help manage this and voice questions if the teacher/lab demonstrator is too busy running the activity.
  • Make sure there is explicit discussion of resources provided on supporting group work and include problem resolution strategies and opportunities.
  • Use polls in the session to check who has viewed the resource and to check understanding of key points
  • In group work, include mechanisms for students to record all their diverse input. Encourage the use of a project tracker such as Trello or a collaborative editing tool such as GoogleDocs, so students can demonstrate their input easily.  Work alongside students to review these records regularly to make sure everyone has an opportunity to contribute.
  • Encourage students to adopt and use their own choice of project and group tools where appropriate. These can provide a valuable record for evaluation as well as being a learning activity in their own right.

At the end of an activity/series of activities:

  • Encourage collective (staff and student) self-evaluation and reflection: “How do we think that went?” “What should we start doing?” “What should be stop doing” “What should we continue doing?” Don’t forget to record this feedback so it can be used to make changes and can be documented within quality enhancement processes. Close the loop and feedback to the students the changes that have been made.
  • Have a strategy for following up on students who don’t seem to be engaging. Use positive questions such as: ‘is there anything preventing you from participating in these sessions?’, rather than ‘why weren’t you here?’
  • Access Blackboard download statistics, EesySoft Course Reports in Blackboard, library resources download statistics. What do they tell you about student engagement?  If you’re not sure, ask the students what they think the numbers mean.
  • Check engagement with published recordings. Ask students to highlight the most useful sections (e.g. there was very useful section on key topic at 5 mins 30 secs).  That will tell you a lot about what the students think they need to learn and what interests them.

At the end of a module/course (summative evaluation)

  • Use a range of evaluative data sources to reflect on the teaching, learning and student engagement for this module/course. These could include learning analytics, student satisfaction surveys, student evaluations of teaching, Student Rep feedback, peer review data and bespoke institutional research and evaluation projects. Make sure you let the students know what the data tells you, and check that it matches their understanding of the success of the sessions (or otherwise).
  • Try not to compare/measure differences in any previous face to face delivery with any blended delivery after March 2020; there are possibly too many compounding variables to make this a worthwhile exercise.
  • Use the quality enhancement processes to evidence actions and impact linked to SMART objective setting.

Institutional evaluation

Published in wonkhe (14.7.20), here are 10 principles for designing and implementing an evaluation of blended teaching and learning at an institutional level.  This would bring together evidence from the activity level outlined above.

  1. The importance of evaluating blended learning should be strategically positioned and systematic, with a shared understanding by all student and staff stakeholders. Aims and objectives should be clear and well known across the institution. The review of evaluation outcomes should be a shared responsibility – and not done in isolation.
  2. An evaluation of blended learning should prioritise, through shared ownership, student involvement in the co-design of the evaluation and promotion of participation/feedback. A communication/dissemination plan should be created and implemented to close the feedback/analysis loop.
  3. An evaluation of blended learning should start with formulated questions – what is being evaluated and why? What does success (quality) look like? These questions should be created by reviewing the evidence base which prompted the change in delivery. Clear definitions of key terms should be provided. This might include institutional benchmarks/thresholds of effective blended learning so a baseline analysis can guide the evaluation and link to the measures of success.
  4. An evaluation should avoid comparing blended learning to a previous face to face approach. Compounding variables and contextual differences make this comparison invalid. This includes comparing data sources over two differentiated time periods (e.g. Student Evaluations of Teaching 20/21 with Student Evaluations of Teaching 19/20)
  5. An evaluation should include formative and summative aspects to allow for multiple points of data collection and a structure for reflection and adaptation of provision (continuous enhancement) as necessary.
  6. The Office for Students’ Standards of Evidence and evaluation guidance should be critically explored at the outset (narrative, empirical, causal evidence) and should feature in decision making.
  7. Evaluations of blended learning should be holistic and include an exploration of:

    a. the learning environment (e.g. technological suitability, ease of use, flexibility, quality of teaching)

    b. the process of learning (interactions with the learning environment via measures of student engagement: behavioural, emotional, cognitive)

    c. learning outcomes: grades and marks, attendance, and withdrawal rates and learning gain (subject knowledge/cognition, personal, generic skills, online skills)

    d. process outcomes (cost, resource, capacity building, support services etc)

  8. An evaluation of blended learning should go beyond institutional data collection and consider the active support of pedagogic evaluation research (e.g. action research), at Course/Dept level. A mixed method/mixed data approach should be used to assess whether the blended approach is effective and why. A range of methods should be employed which utilise existing data collection, adapt existing data collection or are methods which are designed specifically for this evaluation.
  9. A review of the data currently used to evidence quality should be conducted in light of a new model of teaching. As a result, providers should consider the use of validated tools and scales for measuring the effectiveness of blended learning, as new instruments or as additional scales to existing instruments. Data collection should go beyond student experiences to consider the views of a range of stakeholders, including staff reflections.
  10. Evaluation activity should be resourced for all stakeholders, including training and development (for staff and students) to support the synthesis of evidence and the review of evaluation outcomes.

For more information on evaluation and student engagement please contact:

Dr Liz Austen, Head of Evaluation and Research, Sheffield Hallam University l.austen@shu.ac.uk

Dr Jill LeBihan, Head of Student Engagement, Sheffield Hallam University j.lebihan@shu.ac.uk 

Images courtesy of unsplash a, b, c