In this post, Cristina Adriana Alexandru shares findings from a systematic review of virtual classroom software tools. Cristina and colleagues, Kun Zhao and Aurora Constantin, presented at the Learning and Teaching Conference 2022. This post is part of our Learning and Teaching Conference Hot Topic.
The Covid-19 pandemic has forced universities to teach online, then adopt a hybrid approach consisting of a combination of in-person and online teaching. This relies heavily on online teaching solutions, such as virtual classroom software tools (VCSTs). VCSTs are pieces of software which “allow students and instructors to communicate synchronously using features such as audio, video, text chat, interactive whiteboard, and application sharing”, thus making it possible to organise traditional classroom activities online. Some popular VCSTs in Higher Education (HE) are Blackboard Collaborate, Microsoft Teams and Zoom.
While VCSTs may seem a promising solution for online classes, their sheer number, variety of features and design, and lack of systematic evaluations in HE contexts, make it difficult for academics to decide on the best such tool for their HE courses and students. Therefore, the aim of this research was to conduct a systematic review of VCSTs used in HE. In particular, we decided to address the following research questions:
RQ1 : What are the evaluation criteria of VCSTs?
RQ1.1: What are their educational pros?
RQ1.2: What are their educational cons?
RQ1.3: What are the features or design guidelines they should follow to achieve the educational aims by meeting their pros or avoiding their cons?
RQ2: How do current VCSTs compare in terms of these features and design guidelines?
RQ2.1: What is a good framework to evaluate VCSTs?
RQ2.2: What are the best VCSTs resulting from the systematic review?
RQ2.3: What design elements can be added onto the best VCST to improve their functionality and usability?
To address RQ1, we conducted a literature review that culminated in a list of features of VCSTs, which can help achieve their educational pros while avoiding their cons. These features are listed in Figure 1.
To address RQ2.1 we decided to use the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta Analyses) statement, which is frequently used for reporting systematic reviews and provides an evidence-based set of items to this end. However, as PRISMA is typically used for reporting systematic reviews of literature, we had to modify it in order to report the systematic review of software tools. As a result, we planned to review literature only for the purpose of identifying the names of existing VCSTs used in HE, and then interact with each of the identified VCSTs in order to evaluate them. The most important PRISMA items that we used to plan the systematic review were the following:
| Scalability to different numbers of participants | Teams | Multimedia |
| Audio/video conferencing | Breakout rooms | Polling and quizzes |
| Synchronous reactions in meetings | File sharing | Whiteboard |
| Alerts for joining meeting | Capture/record/playback | Participant and list right management in meeting |
| Chat | Subtitles | Ways to access live meeting |
| Application/screen sharing | Integration with internal/eternal tools and services | |
| Meeting scheduling (calendar) | Platform accessibility | |
Figure 1: Beneficial features of VCSTs
- Eligibility criteria:
- For sources (i.e. what are the characteristics of search engine results that make them acceptable sources of VCST tool names?):
- Being in English,
- Reliable (VCST official websites , research or review articles of VCSTs in HE, GitHub pages about projects related to VCSTs, market research websites that list VCSTs and description pages),
- Being free/accessible with university licenses,
- No results which are patents or citations.
- For tools (i.e. what are the characteristics that make tools acceptable to be considered for evaluation purposes?):
- Being in English,
- Having a free trial/ accessible with university licences,
- Being functional,
- Not requiring financial information when registering,
- Updated in the last 2 years,
- Downloadable if on mobile,
- Having a secure account creation/download page (https),
- Having necessary VCST features: live audio and video streaming, screen/application sharing, text chat, whiteboard.
- For sources (i.e. what are the characteristics of search engine results that make them acceptable sources of VCST tool names?):
- Information sources and search strategy. We used Google and Google Scholar as search engines, and (“Virtual Classroom Software Tools” OR “virtual classroom software” OR “remote learning platforms”) AND “higher education” as the keywords.
- Selection process. We first filtered sources by eligibility criteria for sources, identified mentioned VCSTs and filtered them (using official websites and registering/downloading, if needed) according to eligibility criteria for tools.
- Data items (i.e. evaluation criteria). We used the set of beneficial features shown in Figure 1, as broken down into sub-features.
- Data collection, synthesis of results. Each VCST gained 1 point for any data item that was met (i.e. beneficial feature/sub-feature that it had); we then summed up the points per beneficial feature and overall.
Overall, our search in Google and Google Scholar resulted in 717 results (i.e. sources). After applying the eligibility criteria for sources, we could identify 244 VCST names within them. After applying the eligibility criteria for tools, this list was brought down to 28 VCSTs, presented in Figure 2.
Figure 2: Identified VCST names
After conducting the evaluation of the 28 VCSTs using the data items in point 4, we found that the top VCSTs (RQ 2.2) are (in order):
Each of them was remarkable at different features: Teams at meeting scheduling; Adobe Connect at capture/record/playback features; and Blackboard Collaborate at subtitling features. However, each of them fell short in other features, most notably polling and quizzes. Furthermore, eduTinker had the highest score in most features, but only achieved fifth place overall due to shortcomings in meeting scheduling, teams, capture/ record/ playback, subtitles and whiteboard features.
All in all, we could conclude that most VCSTs, even some of the best ones (RQ 2.3), were missing some important features, including:
- Meetings that accept over 200 participants;
- Meeting settings for managing background noise and changing video background;
- Features for inviting teams/sub-teams/ channels to meetings;
- Writing messages within the invitation;
- Displaying invitee availability;
- Teams;
- Quizzes;
- Autogenerated subtitles to recordings;
- Waiting rooms.
We hope that closer consideration of these features, as well as of our detailed results available in the MSc dissertation by Kun Zhao, will contribute to the better (re-)design of VCSTs. This will help academics and universities make informed choices about the VCSTs to use in their courses, and lead to better learning experiences for students.
Cristina Adriana Alexandru
Cristina Adriana Alexandru is a Lecturer in the School of Informatics and a Senior Fellow of the Higher Education Academy. She teaches a large-scale, second year undergraduate Software Engineering course, leads the training programme for teaching support members of staff, organises a summer programming course for incoming first year students, and line manages the team of University Teachers and University Tutors. She is passionate about teaching and building usable and useful technology for teaching and learning, which is why many of her undergraduate, Masters and research projects are in these areas. She hopes these projects will have a real impact on her school and community.