An IFAC Content Series
Assessments are a key feature of the journey to becoming a professional accountant. Every year, thousands of aspiring professional accountants write low- to high-stakes in-person paper-based or computer-based examinations.
Until recently, very few examinations have been taken remotely. But with COVID-19 forcing many examination centers across the world to close, professional accountancy organizations (PAOs) have had to either postpone or pivot to remote online examinations. PAOs that have already embarked on this journey (albeit at a slower pace), with a higher risk tolerance, and with the necessary resources were able to transition more rapidly to remote online examinations. This required agility, a high level of collaboration amongst key stakeholders, and an adaptive mindset—or in short: blood, sweat, and tears.
Whether they pivoted or postponed, most PAOs agree that the landscape has changed forever: there is no turning back; remote online examinations are the future. Those that have pivoted are eager to enhance their processes and systems, and those that have postponed are eager to learn from those who have pivoted. This was evident from a recent Virtual Knowledge Sharing event hosted by IFAC. The event was attended by more than sixty individuals representing PAOs, accounting firms, and universities from approximately 25 countries.
Alan Hatfield of the Association of Chartered Certified Accountants (ACCA), Ronan O’Loughlin of the Chartered Accountants Ireland (CAI), and Simon Hann of the Chartered Accountants Australia and New Zealand (CA ANZ) shared what is possible—and necessary.
With the International Education Standard requirement for PAOs to design reliable, valid, equitable, transparent, and sufficient assessments as backdrop, we asked the participants to consider a few questions.
How does the type of assessment affect the transition from in-person to remote examinations?
Participants supported a phased approach in which risks and resources are managed by starting with smaller groups and low-stakes examinations before moving to larger groups and high-stakes examinations, while continuously adapting to any outcomes. They felt that although this makes for a longer transition, it will be worth the effort.
Navigating case studies and answering essay-style questions in an online examination appeared to be challenging, and some participants questioned the effectiveness of oral assessments via online webinar platforms. Proposed solutions included combining long- and short-form questions and oral assessments or distributing case studies before the examination.
Participants discussed the use of objective testing for low-stakes assessments and the ease of marking this type of assessment by computer. Those who were concerned about perceptions that objective testing lacks credibility asked where low-stakes assessments end and high-stakes assessments start, as they sought to maintain the balance between cost and credibility. Others questioned whether high-stakes assessments would continue to be the preferred approach, arguing that papers and hours do not have to be long to be credible.
Monitoring the use of a one- to two-page note, often allowed for in-person examinations, created a dilemma for remote online examinations. One PAO overcame the problem by allowing open-book examinations.
Everyone agreed that the examination should be designed for the format and platform through which it will be delivered. Many were excited about computer-based examinations mirroring work experience and promoting workplace skills.
What changes must be made for a successful transition?
Stakeholder skepticism appeared to be a key challenge. Questions about academic rigor and integrity, security, and assessment credibility in a remote online examination environment have arisen. Some participants spoke about the effort and time necessary to obtain approval at different levels of their PAOs’ governance structures, and the importance of brining employers and regulators on the journey, while others found their regulators more open to change in the current environment than previously.
The use of biometric data to verify student identity (such as key stroke verification) can be a challenge considering the European Union’s General Data Protection Regulation (GDPR). Given these concerns, a PAO chose to rely on a human proctor with students presenting official documentation as the way to confirm their identity. Data privacy is also a challenge in the case of remote proctoring where students are required to turn on their cameras during the examination. In at least one case, students are required to have two cameras: one facing the student and another the environment around the student. One university responded to this by asking students to accept this requirement at the time of registering for a course.
All agreed that the transition requires a change in the mindset of all actors in the examination chain—students, PAO staff, educators, examiners, proctors, markers, et al. Training and guidelines are essential and appeared to be lacking in cases of rapid transition. Surprisingly, some PAOs found that, although students are proficient in using their phones, they lack keyboard skills. Offering them resources for typing practice is a good way to overcome this challenge.
Work remains to be done in marking online assessments less suitable for computer marking, such as objective testing. A PAO that used to gather a team at a physical location for a few days to mark papers transitioned to Microsoft Teams to create a similar environment for remote marking. The markers were in agreement that not having to decipher students’ handwriting is a huge upside of computer-based examinations.
Participants discussed the importance of supporting students for remote examinations just as well as for in-person examinations. Some saw higher stress levels and panic when students take high-stakes examinations remotely. Students need support during their first remote examination. Our participants floated virtual sessions to review the examination guidelines with students, practice (mock) examinations, guidance on how to overcome hardware problems such as connectivity issues, and call centers with enough well-trained staff to handle any problem—including mental health. A participant suggested that PAOs planning to transition after COVID-19 hold their first remotely-proctored examination in centers to offer students a glimpse of the future.
No doubt, good practices will emerge over the next few years as PAOs continue to use their learnings to modify their remote online examinations, and more complex matters such as additional time for connectivity issues, re-sitting in case of malfunctions, accommodations for disability, comfort breaks, et cetera will be addressed.
What is the impact on resources?
Transitioning to remote online examinations is expensive in terms of people, processes, and systems, and will require a multi-year investment. Nevertheless, those who have embarked on this journey are of the view that, over the long term, the benefits will outweigh the cost.
A robust information technology infrastructure is essential. All actors in the examination chain have to be trained and supported. Papers have to be redesigned. And sometimes the investment is not only by the PAO but also by students, as not all students have equal access to the necessary hardware or software or to an environment that is conducive to taking examinations. Nor are all students equally prepared to handle so much change in the transition to remote examinations.
Participants shared measures to cut costs. For example, a university that temporarily transitioned to remote examinations for low-stakes assessments decided not to invest in remote proctoring but rather communicate to their students that they trust them. A few PAOs shared how they are strengthening their question banks affordably by using “cloning” software.
Participants from smaller PAOs inquired about regional solutions. They suggested that the move to common content creates an opportunity for common assessment, while also recognizing that language might be a barrier. Some asked for a list of PAOs that have progressed on the journey and would be willing to share good practices or provide advice or assistance. At least one participant’s PAO was using the system of another PAO while developing a small number of assessments covering local content.
What technology is being used and how is it different for different types of examination?
There was clear agreement that the technology should contribute to, not detract from, the quality of the examination. PAOs that have embarked on the journey are using a variety of platforms and—in most instances—different platforms for writing, delivering, proctoring, and marking. Where different platforms are used for different aspects, integration of platforms is essential. Everyone agreed, PAOs will benefit from the creation of generally accepted criteria to consider when selecting a platform or platforms, as well as sharing of experience by those that have made progress already.
How should the change be communicated and managed?
It was clear that communicating and managing the change have been challenging—particularly in cases of rapid transition.
All participants agreed that the basic principles of effective communication apply: the imperative to use relevant, brief, clear, consistent, timely, and targeted language. Participants recommended the development of a comprehensive campaign-like communications plan that would bring a wide range of stakeholders into the process. For students, they suggested overcommunication, streamlined messages, infographics and videos, and a combination of various means of communication, including email, text, WhatsApp, and telephone calls. A student’s first remote examination is exceptionally stressful; PAOs should prepare for an unusually high level of questions from students about the examination. But it was also noted that students generally tolerate and understand this change.
Transitioning from in-person to remote online examinations requires a change in mindset. That takes time. And even though the appetite for change has been greater in the current environment, it has not always matched the appetite for risk. A PAO shared that it strengthened the case for change made to its board by incorporating the voice of current and future members and employers. This PAO emphasized the need for relevance and being flexible to employer needs.
PAOs that are considering transitioning were advised to develop a comprehensive change management plan that covers a wide range of stakeholders—internal and external—and to identify champions to drive the change. The importance of investing in the training of stakeholders was emphasized. Involving stakeholders, including employers and regulators, early, seeking their feedback along the way, and being flexible in accommodating their needs will help to secure their buy-in.
It is clear, one size does not fit all. PAOs were encouraged to pursue solutions tailored to their resources.
We encourage you to continue the conversation
Do you have additional challenges, solutions, or opportunities? Do you have policies, procedures, guidance, et cetera that might be useful to PAOs? Are you willing to assist a PAO on the journey to online examinations? If your answer to any of these questions is “yes,” please contact us at education@ifac.org as we plan to continue to share your knowledge in the Accountancy Education eNews (subscribe today!) and link to your good practices in the IFAC Accountancy Education E-Tool.
Don’t miss these case studies:
