ChatGPT and accountants have battled it out. And, the AI is proving to be no match for human beings when it comes to accounting. Accountants! WE RIDE AT DAWN.
Since it first fired up in November 2022, ChatGPT, owned by OpenAI, has become the fastest-growing technology platform ever. It reached 100 million users in just under two months.
OpenAI recently launched its newest AI chatbot, called GPT-4. This iteration of the bot managed to pass the bar exam with a score in the 90th percentile. Researchers found it was also capable of performing “close to the passing threshold” when tasked with the three-tiered United Stated Medical Licensing Exam.
But what about other professions? Researchers at Brigham Young University (BYU) – in collaboration with personnel from 186 other universities – decided to find out by testing ChatGPT in the accounting field. So are accountants going to be obsolete?
ChatGPT and accountants
The lead study author, David Wood, is a BYU professor of accounting. Speaking to BYU’s website, Wood stated “When this technology first came out, everyone was worried that students could now use it to cheat. But opportunities to cheat have always existed. So we’re trying to focus on what we can do with this technology now, that we couldn’t do before, to improve the teaching process for faculty and the learning process for students. Testing it out was eye-opening.”
To work out how such AI chatbots should factor into education, Wood decided to recruit professors to help test the AI against accounting students.
His recruiting pitch on social media totally blew up: 186 educational institutions in 14 countries participated. They also pulled in BYU students to help feed test questions to the AI chatbot. The questions covered accounting information systems (AIS), financial accounting, auditing, managerial accounting and tax, among other things.
ChatGPT’s performance was impressive, but even despite this, the students performed better, scoring an average of 76.7%, compared to ChatGPT’s 47.4%.
ChatGPT is not a star accountant
On 11.3% of test questions, the AI chatbot scored higher than the average student, doing well on AIS and auditing. But ChatGPT did worse on tax, financial assessment, and managerial assessments.
The AI chatbot did better on true/false test questions (68.7% correct), and fared well on multiple-choice questions (59.5%). However, it struggled with short-answer questions (between 28.7% and 39.1%).
In general, more complicated questions were difficult for the AI chatbot to answer. The researchers even report that “sometimes ChatGPT would provide authoritative written descriptions for incorrect answers.”
Jessica Wood, a freshman at BYU, said, “It’s not perfect; you’re not going to be using it for everything. Trying to learn solely by using ChatGPT is a fool’s errand.”
The researchers tasked with testing ChatGPT’s limits discovered a number of core problems:
- The AI chatbot doesn’t always realise when it is doing math. It makes nonsensical errors, like adding two numbers in a subtraction problem. It also sometimes divided numbers incorrectly.
- The AI chatbot often provides further explanations for its answers that are totally incorrect.
- When ChatGPT’s descriptions are right, it can still go on to select a wrong multiple-choice answer.
- ChatGPT sometimes totally makes up facts. It can generate a real-looking reference, but it is completely fabricated. The work itself and sometimes the authors have never existed.
The verdict
While human accountants still outperform ChatGPT, there will come a day where AI does things better and faster. The researchers expect GPT-4 to improve exponentially on the accounting questions in their study, and in the future, not have the snafus above.
Accountants have surprised us yet again, not only by keeping us safe from tax mistakes, but also by outsmarting AI chatbots (for now). Carry on, accountants.
