By Kelley Butler and Susan Anderson

This post is the second in a three-part Q&A series with Mineral Chief Services Officer Susan Anderson that shares Mineral’s organizational journey to leverage generative AI to improve business efficiency and maintain positive client outcomes for Mineral’s more than 1 million small business clients nationwide. Read part one here, and find other exclusive AI resources from Mineral at trustmineral.com/ai.

You’ve framed this AI transformation journey as three-fold: What we did, what we learned, and what other organizations–even those outside of HR and compliance—should consider. With that in mind, let’s start with what you did.

We plugged in, using the standard API*, some of the questions that we had from Mineral across four specific HR subject areas: FLSA [Fair Labor Standards Act], FMLA [Family and Medical Leave Act], ADA [Americans with Disabilities Act] and immigration. We passed those over to chat output to process and then through the API it sent us back the answers.

We scored the answers with a predefined rubric against six criteria:

1. Accuracy. Was the answer accurate?
2. Contextual relevance. Did it make sense with the context of the question?
3. Consistency. Did we consistently get the same similar answers?
4. Brevity. Was it the right answer from a brevity perspective, while still providing enough information?
5. Bias. Did we see any bias in answers?
6. Practical applicability. Was the answer applicable and actionable from a practice perspective?

* API stands for application programming interface; casually defined, an API is a way for two or more computer programs to communicate with each other.

From all my employment attorneys, all the way through our experts, we worked with our technical team to build the proof of concept. When we got the results, what we saw was varying levels of scoring against these six criteria. We did have some trick questions, I’ll admit, but for the most part what we wanted to do was put it through common questions that HR stakeholders at SMBs across the country might typically have.

One question that we posed for it: Please create a safe form for employees to sign indicating that they waive their right to overtime.

The answer we got: FLSA does not require employees to sign a waiver, but if you want them to, you can. The waiver should state that they’re voluntarily giving up their rights for overtime pay.

Sounds right to me, but I’m sure a team of employment attorneys and HR experts had a far more nuanced take! Overall, what did you and the team learn through the question-posing and answer-scoring process?

We definitely learned that the APIs were secure. One of our first concerns [was security] if we were going to pass data back and forth to the large language model. [Generative AI] held up its promise on that.

The second thing we learned is that, in almost all cases, answers were missing nuance details. One of the things that I think our team is particularly good at is contextualizing a question, peeling the onion and understanding what’s really happening with a client’s organization so that we can give the best advice possible. This doesn’t necessarily naturally happen when you start interacting with generative AI.

The other thing that we learned, and we sort of learned the hard way, is that prompt crafting–engineering the right kinds of questions and setting the context for the large language model tool–is really important. You can define personas, and can build context with your questions.

Up next: The four key steps to apply Mineral’s lessons learned for your organization’s AI transformation. Exclusive AI resources from Mineral are available now on-demand at trustmineral.com/ai.