Evaluating the IVA Experience, Part II: Machine-Driven Analysis

Recently, we covered part one of evaluating your intelligent virtual assistant (IVA). We talked about framing your evaluation based on business goals and your user’s needs, guided by the question: “Given what I know about my business and my customers, did the IVA provide as much value as possible?” This is the first 180 degrees of evaluating your IVA. Let’s get to 360.

IVA’s are continuously listening and learning from your end-users to refine conversational skills, absorb concepts, and optimize responses. However, the constant flux and evolution of language, end-user preferences, and business goals mean your IVA is operating in a dynamic environment. In these fluid conditions, how do we determine if the IVA is consistently understanding and successfully responding to the end-user?

If an IVA is handling 200,000 user engagements per day, that means it’s trying to achieve 200,000 unique versions of successful interaction.

Realistically, evaluating performance at that scale requires wrangling more data than any human could reasonably manage by hand.

Enter: automated analysis.

At Next IT, every IVA we deploy for customers comes with a set of proprietary tools that helps businesses fine-tune their IVAs at massive enterprise scale. At their core, they are automated technologies that identify areas that may need improvement, that uncover specific problems, and that curate comprehensive solutions.

Alme’s analysis tools start by consuming and analyzing vast quantities of chat data, identifying instances in conversations with the IVA where the experience could potentially be improved. These indicators can range from the user expressing frustration or trying to restate their question, to automated learning systems recommending a better outcome.

Let’s say we have 200,000 interactions – our system searches for these indicators in each of them and automatically surfaces gaps in the IVA’s understanding. The information is then sorted by topic and severity, based on your business. Just as your IVA is purpose-built to address your business goals, Next IT’s analysis tools allow you to focus your evaluation on an area that will provide the biggest impact for your business and users.

Once we know where things are going wrong, our tools analyze the data to help unlock the problem. They can process large amounts of natural language inputs – all of that chat – and help define the cause of the misunderstanding. Which language inputs are causing the most problems? Is there unclear messaging or content that’s causing a disconnect? Is there an opportunity to develop a new product or service? You’ll get a picture (in your customers’ own words) of what you’re doing well, where you can improve, and what they need from you. These insights can be used to improve the information you make available, products you sell, and services you provide.

Whether you discover a need for new systems of information, changes to existing systems, or modifications to the language processing model, our analysis tools help you create a comprehensive solution that will lift business performance and make it easier for users to get what they need. As a Next IT customer, these proprietary tools empower you act on these insights and validate the changes that serve a distinct purpose for your enterprise goals.

We’ve come full circle in the human/machine conversation – this 360-degree holistic approach to evaluation gives an enterprise the dexterity they need to accommodate constant change in their business, while also creating room for human-based evaluation that leads to creative insights.