As the largest accountancy body on the island of Ireland numbering 29,000 members, Chartered Accountants Ireland is proud of its mandate as the sole education provider and examining body for its 6,600 Chartered students. Back in February 2020, the future of our examinations looked steady, with a tried and tested model based on traditional exam halls, invigilators, paper scripts and exam markers huddled together in dense concentration over the following weekends. What could possibly go wrong?
By that stage, the exams team and leadership in Chartered Accountants Ireland had amassed considerable experience in new examination developments and adapting to changing educational demands, which would become a distinct advantage. New additions over recent years meant that the Institute was change-ready when the time came. As its strategic plan has “digital first” as a guiding principle, some minor interim exams were already slated to go online from November 2019 with the pilot finishing in April 2020, covering about 10% of all assessments.
On Thursday, 12 March 2020, the Taoiseach (Irish Prime Minister) appeared live on TV to announce a lockdown for business and society, with the devolved government in Northern Ireland following suit soon afterwards. In the face of such uncertainty, the challenge set to the Institute was to ensure some form of exams to support the progression of students and the firms’ training model, in a way that minimised the health risk to students, staff and other parties.
For the Institute, initially it appeared that rescheduled exams in mid-August was a long way away and the main exams might well carry on as normal, and an E-Assessment alternative was developed as a back-up plan. The first step taken was to postpone exam dates, and this was communicated to all students. However, as the public health crisis escalated, the notion of students travelling to sit exams together in a traditional setting was soon out of the question.
By May, it was apparent that the back-stop option had become the only option, and Institutes in Australia and New Zealand moved to place their professional exams online. Irish plans for a two-year, gradual move to online-only exams had to be rapidly accelerated.
If exams were to go online, the profession’s regulators, training firm partners and the Institute were all determined to ensure the exam hall experience would be replicated in an online environment insofar as possible. This meant ensuring the students could prove their identity, that they were alone during the exams, did not have any notes (unless the exam was “open book”) and mobile phones had to be out of reach. The Institute had already assigned international invigilation providers ProctorU to provide remote services for its April 2020 interim assessment and extended the arrangement for all main exams. In addition, a new bespoke exams platform was commissioned and developed by online exam providers Cirrus.
Planning for the April 2020 interim exams for first-year students was used to good effect and rapidly scaled up for main exams in August and September, with many important lessons learned and issues addressed. A second special repeat interim exam had been organised as a precautionary measure and ended up being offered to a number of students. As a result, special sittings in case of unforeseen technical issues were included in the planning process for the August exams, to take place two weeks after the main exams. This provision created a huge additional workload for the examination development team, whose output of test, main exam and special sitting exam papers almost doubled.
The basic exams day process for students in April was carried forward into the main exams in Mid-August. Students were sent an initial link to commence the onboarding process. The student was then connected in-person with a remote proctor via the laptop camera, who took the student through a validation process and checking their immediate environment. Once the checks were complete the student could enter the exams platform. The exam time began once the proctor started the onboarding process.
Complicating factors
While this all appears straight-forward, in reality there were multiple complicating factors. With the exception of those in first year who sat interim assessments, students were entering professional exams in unknown territory, and these would be the first meaningful online exams they had ever sat.
The viability of online exams depends to a huge degree on the strength and consistency of broadband and Wi-Fi signal, something that was clearly outside the Institute’s control. In Ireland, many locations outside urban areas have weak or intermittent broadband, perfectly fine for work or leisure but insufficient to maintain an exam link for up to four hours.
Most students relied on their own laptops for the exams, some of which were either old, had many applications running in the background consuming broadband or had some malware or very limited performance speeds. All of these would have caused issues for students either to access the exam or to drop out of the exam repeatedly.
Where students were offered a laptop by their employer or offered an office location to sit the exam, employer firewalls often denied access to the exam platform. And the ProctorU interface required frequent updates to ensure the latest version of Chrome internet browser was in use.
Training and support
To address as many of these issues as possible, it is crucial to provide as clear a picture as possible of what to expect and how to avoid problems and to stay connected with stakeholders. A wide range of training materials and resources were produced quickly and issued to students, including training videos, exam questions and solutions and sample papers to complete on the new platform.
Students also had the chance to join a trial onboarding process to help assess the suitability of their laptops and web connectivity and to resolve any technical issues. Students were further provided with up to five practice papers to complete. Regrettably, there was only limited engagement with these support services, with student take-up a week before exams at around 30%.
Close contact with the larger training firms was maintained, and key employer stakeholders contributed.
Intensive support services for students on exam day were anticipated and prepared. A process to deal with exam-day problems was designed to enable the team to manage and escalate responses as needed. We had asked students to familarise themselves with the response sequence and this was largely adhered to.
Performance of online platform
Taken as a whole, the new E-Assessment platform, the online invigilation and associated processes all progressed extremely well, with average completion rates of 98.6% for the first set of main exams for our second year students, 99.6% for final year students and 99.5% for first year students. These are the highest rates achieved by peer institutes and is some way ahead of that achieved by other accounting bodies.
Despite herculean efforts, of course there were some issues encountered and not all students had a faultless experience. The main issues causing problems were student-side broadband interruptions of varying degrees (80%) and firewall or hardware issues (10% each).
It was to be expected that any difficulties would arise on the first day, and so it transpired, with a minority of students impacted with delays or some interruptions. In the main, these issues were addressed in time for the remaining exams that week. Where a small minority encountered serious faults, students were offered a place on special resit exams two weeks afterwards and this resolution mechanism worked well. Some students indicated that the typing and navigation functions within the exam platform were somewhat cumbersome and this will be refined in future. Overall, over the main sittings and additional special sittings 100% of students completed their exams on the new platform.
Our Top 10 Tips:
- Remember this is a change management project so engage with all stakeholders over the project.
- Understand your regulator’s concerns early on so these are fully accommodated.
- Be proactive, plan ahead and be flexible and adaptable.
- Increase your engagement with your partners to ensure high onboarding success.
- If students use secure office laptops you will need to resolve and get approval for Chrome extension or equivalent.
- Bringing your students on the journey is key so provide appropriate briefings and learning support services for students and work with your firms and partners to drive engagement. Consider a student E-Assessment code of conduct.
- Look after your own team. They are carrying a significant responsibility and workload and they need to be supported and recognised.
- Put additional students and technical support services on the day of the exam—things always turn out differently than you expect.
- Test, test, test ...
- Keep your internal governance informed of progress and issues. If this is working well, they will be helpful in maintaining momentum and credibility.
Conclusion
Over 6,600 Institute students have just completed professional exams online for the first time in the Institute’s history with almost all students succeeding in completing their papers, all while protecting students’ health and acceding with government health policies. A new platform that started as an unlikely contingency measure back in April successfully shouldered a full programme of professional exams a mere four months later, and we have established that we have a viable online exam platform at the ready, should it be required.
