Can ChatGPT Really Replace HR Pros? New Study Puts it to the Test
HR can’t seem to escape ChatGPT and other AI technology.
Although some HR pros fear that AI will completely overtake their jobs, others can see the benefits of utilizing AI to help them do their jobs better. But no matter which side you’re on, the question remains: Can AI really do the job of an HR pro?
A new study conducted by Mineral put that question to the test, asking ChatGPT several essential HR questions. Here’s what they found.
Putting it to the test
Mineral’s test used several versions of ChatGPT – 3.0, 3.5 and 4.0 – to ask several HR questions, covering topics from termination and time off to discrimination and salary transparency. Questions ranged in complexity, but they covered essential HR topics like FLSA and FMLA.
Specifically, the AI was asked:
- What is “at-will employment”? Does that mean I can fire an employee for any reason?
- I know that most of the federal employment discrimination laws kick in when an employer reaches a certain number of employees. How do we count that number?
- How do I calculate my employee’s FMLA leave?
- What is EPSL and EFMLA? Are they required?
- What salary range should I post to meet pay transparency requirements?
- Then, Mineral got six HR pros – five HR experts and one employment law attorney – to rate the answers on a scale of one to 10 based on factors like accuracy, relevancy, consistency, brevity and practical applicability.
Can AI replace real HR pros?
The results were clear: “ChatGPT is not here to replace skilled HR practitioners,” says Susan Anderson, chief services officer at Mineral. “The answers it provides are too often inaccurate, incomplete or misleading in a way that could be costly to a business that wants to lean on the tool to answer HR questions.”
While the AI could handle linear, straightforward questions, its answers became unreliable as questions got more complex. The answers are also limited to information available in 2021 or before, so the accuracy of the answers fell when new information came into play.
The answers weren’t just outdated, though. They also had significant inaccuracies. “In almost all cases, the answers ChatGPT provided were missing necessary nuanced details and in some cases ChatGPT completely fabricated reference material that did not exist at all,” says Anderson.
For example, when asked, “What is at-will employment? Does that mean I can fire an employee for any reason?” The AI states that an employer could fire an at-will employee without “cause, notice or consequence.” The use of the word “consequence” could be misleading, especially to someone who isn’t familiar with the world of HR.
“An untrained user with little or no HR experience could easily be misled by a ChatGPT answer that sounds accurate, while a trained practitioner will recognize that the answer is incorrect or that key details were left out,” says Anderson.
How AI will change HR
Despite the pitfalls of some of the answers, the study still reveals that AI tools can be valuable tools for HR pros. “ChatGPT can and will change how HR is practiced by streamlining certain processes,” says Anderson. “Taking ChatGPT’s answers and combining them with an HR professional’s knowledge can help get accurate and reliable answers in far less time.”
However, there are still disadvantages that HR pros need to be on the lookout for when utilizing AI for HR questions. “One common pitfall of ChatGPT is that it does not consistently generate answers to questions with the same facts and details every time,” says Anderson. “One answer may lack the specificity needed to properly answer the question and other answers may lack necessary information based on when that information became available.”