Lessons learned from a GenAI proof of concept

More companies are experimenting with generative AI (GenAI). How they approach and learn from early proofs of concept (POCs) can make a huge difference in successfully deploying the technology across the enterprise.

A leading international infrastructure company, for example, sought our expertise to better understand if GenAI could improve the accuracy and efficiency of its work order management system. The company, which manages housing complexes across the U.S., tracks and responds to 500,000-plus annual maintenance requests. Tasks include scheduling, dispatching, updating and solving service-related problems. Roughly 70 employees manually process about 1,500 work orders per day. The team uses the information and notes that operators and/or technicians enter in the system. This largely manual approach creates a sizable opportunity for error and variances in how each work order is categorized and managed.

In under two weeks, we developed a GenAI-powered solution with the client’s current challenges and its goal of using what was learned for process improvements in mind.

Defining the POC scope and strategy

The goal for this POC was to determine whether an AI-powered solution could deliver more accurate results and greater consistency across the board. For the sake of time and to demonstrate the impact, NTT DATA and the client determined the POC would focus on three main business questions:

1. What expertise is required to address the resident’s problem (classification)?
2. What is the urgency level for this request (time prioritization)?
3. Are there any special-handling requirements (for example, the resident wasn’t home, a return trip is needed or parts need to be ordered)?

The client also asked that the POC be built and hosted in a secure, third-party environment with a privately hosted large language model (LLM) — in this case, NTT DATA’s. Doing so mitigated risk and demonstrated that the fully functional AI tool could be built out within the client’s firewall.

Building the knowledge base

The client had well-documented policies, procedures, requirements and a comprehensive list of over 160 work order categories. Our team used this information to prompt the LLM on the intricacies of accurately classifying incoming requests. This approach provided reasonably accurate output for work order requests that even human operators could get wrong. For example, identifying a reported leak as a plumbing or HVAC issue. The output also qualified urgency level and interpreted whether a request required special handling.

Long-term, the client will use a custom application to adjust its policies and add clarifications for the LLM, so the process will continue to improve over time.

How GenAI makes a difference to the solution

Our work order management POC used GenAI to provide the reasoning behind the provided classifications. So, the LLM not only classifies each work order before moving on to the next but explains why each one was categorized the way it was. This transparency into the “why” provides a valuable audit trail that enhances accountability and provides insights into how the company’s current manual process can be improved. The generative AI solution demonstrated the capability to more quickly, accurately and consistently classify work orders than the current, largely manual approach. Over time, the LLM will continue to improve as policies are tweaked and provide superior consistency against the performance of human operators (with little to no additional training).

Implementing GenAI in the real world

Completing the GenAI POC helped our client see the potential of this tool and how it can benefit the work order process — and ultimately improve residents’ satisfaction and safety. In the near term, the company plans to use the solution to understand how best to train and equip its current work-order processing team to optimize performance and improve outcomes. Once the LLM consistently demonstrates its usefulness, the company will plan a full deployment.

Key lessons learned from the GenAI POC

• Success with GenAI requires a balance of business insight and technology capabilities. Start with business value, then examine technology enablement. Identifying and prioritizing the right POCs is the first step in any GenAI initiative. For our infrastructure company client, knowing the work order management POC had strong potential to improve efficiency and resident/client satisfaction made trying GenAI an easy decision.
• Move fast, but be patient. Early GenAI POCs may not prove their value the first time, or they may reveal a different path forward. Tackle your POCs with speed, so you can quickly test their sea legs, but be patient enough to learn from your efforts and try again if the first use case doesn’t live up to expectations.
• Don’t go it alone. With a new technology like GenAI, you likely won’t have all the skills you need internally or the perspective to know how and where it can have the most impact on your organization. A partner can help you evaluate the GenAI marketplace and focus on the best approach for your business. That said, choose your partner wisely. Don’t focus solely on acquiring the skills you don’t have; use your partner’s experience and insight to help you identify ideas or iterations beyond what you identify internally. And ideally find one that’s willing to share risk and return in an environment as fast-moving and innovative as our current one.

Learn more about how our team can help your organization along on its AI journey.