Overview

We built GenAI solution for an award-winning solutions provider based out of USA for supply chain management

In the dynamic landscape of warehouse management, the ability to swiftly access and analyze crucial data can make all the difference.

Tarka Labs, at the forefront of GenAI solutions, embarked on a transformative journey with client name to redefine how organizations interact with their data warehouses. This case study delves into the intricacies of leveraging Large Language Models (LLMs) to create sophisticated chatbots capable of revolutionizing warehouse management.

Challenges

Getting the right information at the right time is not easy

The warehouse employees generally encounter several challenges that hinder their operational efficiency. These challenges, which prompted the exploration of advanced AI solutions, can be outlined as follows:

Limited Data Accessibility:

• Warehouse employees faced constraints in accessing and extracting pertinent information from the data warehouse.
• The process of retrieving data, especially related to incoming shipments, historical sales, inventory status, and open orders, was time-consuming and cumbersome.

Dependency on Dashboard Development:

• Traditionally, the power to access new information is centralized in the hands of the IT team responsible for planning and building dashboards.
• Business users face challenges in dynamically requesting and receiving real-time information without the need for constant IT intervention.

Objective

Integrate GenAI ChatBot with the warehouse management system

This exploration aimed to empower employees within organizations that handle large volumes of data. The goal was to enable these employees to extract precise information from their data warehouse management systems. Natural language queries about incoming shipments, historical sales, inventory status, open orders, and supply forecasting became the focal point.

Tarka Labs aimed to showcase how these AI-driven chatbots could revolutionize the way organizations access and analyze their warehouse data.

Approach

Leverage GenAI LLM model to write complex SQL queries

Tarka Labs adopted a systematic approach, experimenting with various LLMs to gauge their accuracy, understanding of warehouse concepts, and suitability for handling sensitive data and ability to generate SQL queries.

List of sample questions used for POC and how each LLM fared with it

Solution

ChatGPT4

In the exploration of advanced AI solutions for warehouse management, ChatGPT emerged as a pivotal tool, showcasing remarkable accuracy. Through a systematic two-step process, ChatGPT excelled in responding to 9 out of 10 sample questions.

Its initial step involved identifying pertinent information from the data warehouse management system, leveraging inherent knowledge of warehouse management terminologies. Subsequently, ChatGPT seamlessly crafted queries, demonstrating a nuanced understanding of the warehouse domain.

The implementation further benefited from prompt tuning, where pre-defined definitions were provided to enhance response accuracy, highlighting ChatGPT's adaptability and proficiency in the warehouse management landscape.

Technical Approach

The success of ChatGPT in warehouse management queries was attributed to a strategic combination of tools, incorporating agent-based function calling and leveraging the LLM's SQL generation capabilities.

This systematic approach aimed to address challenges methodically, ensuring precision in responses.

Analyzing Question and Context:

• The process commenced with a thorough analysis of the user's question and the contextual information available.
• This analysis facilitated the creation of a well-structured SQL query, integrating it into the reasoning process.

Utilizing SQL Runner Tool:

• The generated SQL query was seamlessly executed using our custom built SQL runner tool.
• This tool not only executed queries but also featured a self-healing mechanism, ensuring robust performance even in the face of malfunctioning queries.

Determining Response:

• Upon executing the query, the LLM determined the course of action based on the received data.
• If additional information was required, the system iteratively revisited steps one and two.

Calculations and Formatting:

• With sufficient input, the LLM performed necessary calculations and meticulously formatted the response.
• The formatted response was then returned to the user interface, ensuring a seamless and user-friendly interaction.

Prompt Tuning for Enhanced Accuracy

To further enhance response accuracy, the implementation incorporated prompt tuning. Pre-defined definitions were systematically provided to the LLM, enabling a more nuanced understanding of the warehouse management domain. This adaptability and proficiency positioned ChatGPT as a valuable asset in the evolving landscape of warehouse management, showcasing its transformative potential in addressing complex queries with precision.

SQLCoder

SQLCoder-34B, an open-source Language Model (LLM), was explored due to privacy concerns surrounding GPT4, as our client handles sensitive data. This 34B parameter model specializes in generating SQL queries from natural language and is fine-tuned on a base CodeLlama model, incorporating over 20,000 human-curated questions across ten different domains.

However, SQLCoder-34B faced challenges in its functional performance. It struggled to identify relevant tables for addressing questions, managing to answer only 2 out of 10 questions. Additionally, the model couldn't efficiently break down complex queries to provide solutions.

Technical Approach

From a technical standpoint, Amazon Sagemaker was employed to deploy this model. Given its fine-tuning for SQL responses, SQLCoder-34B generates SQL output. Despite producing syntactically correct queries, the model's limitation surfaced – lacking fine-tuning for warehouse-specific queries resulted in logically incorrect queries

Llama2 70B

Llama2 70B, with its solid grasp of warehouse concepts, was investigated for its proficiency in SQL generation. Employing prompt engineering and few-shot learning, we guided the model through prompts to generate SQL for specific queries. We utilized a pre-trained model without fine-tuning for this.

While Llama2 70B demonstrated adept instruction following, it struggled to identify the relevant information for answering questions. The queries it produced were syntactically incorrect, and it used inaccurate column names within the data warehouse. The model generated hallucinated SQL queries, referencing random columns not present in the prompt.

Technical Approach

We tweaked top_p and temperature parameters and the responses were generated accordingly. Despite its shortcomings in SQL generation, Llama2 70B exhibited commendable reasoning capabilities, providing clear explanations of the logic used in SQL generation.

Hybrid Approach

Recognizing the strength of ChatGPT4, a hybrid approach was proposed. This involved blending the reasoning capabilities of ChatGPT4 with an open-source LLM.

This strategic combination aimed to generate accurate responses while preserving the privacy of sensitive data. By exposing only the schema to ChatGPT4 and executing queries in an SQL engine, a harmonious balance between accuracy and data security was achieved.

Conclusion

GenAI capabilities can be tailored for any organisation to give them the edge

In conclusion, Tarka Labs' journey with client name underscores the transformative potential of AI-driven chatbots in warehouse management.

Beyond the technicalities, this case study serves as an invitation for organizations to explore the untapped potential of LLMs.

As technology continues to evolve, leveraging these tools to their full capacity can unlock new possibilities and redefine the way we approach data in the warehouse management landscape.