AI was my Worst Agent #1 - Item Selection
AI was my Worst Agent #1 - Item Selection
Journaling the challenges and solutions in making AI the primary support agent in a production environment.
MelodyArc is an operations platform that uses both AI and human agents to service users. AI are the primary workers, with human agents available for support. You can learn more about our platform here.
Using AI as the primary worker in a production environment is hard. This series shares our journey in solving specific challenges.
The Task
For retail e-commerce customer support, when a customer is asking for order support, the Agent needs to identify which items a customer is referencing.
Requirements
Agent must correctly identify all items referenced by the customer.
Agent must respond in a pre-defined format.
🔔 In live operations, an AI agent can ask a human agent for help if it is not confident which items to select.
Round 1
To begin, we have chosen ChatGPT 4 as the AI to power our agent. We will start with a straightforward task: developing a response mechanism using a simple customer message and order set. Our focus will be on defining how data is presented to the AI and how it interprets and responds to this data
🔔MelodyArc AI agents run using APIs. Results can differ when using ChatGPT APIs vs the UX.
Prompt
Your Role: As an expert customer service agent, your primary task is to interpret customer emails and deduce the referenced items from the details given, creating a response in the form of JSON objects.
Instructions:
1. Analyze the Message: Carefully read the provided customer message. Read the entire message before proceeding.
2. Analyze available Item options: Items that the customer ordered will be presented in this format.
"Order Number \${order_id} placed on \${placed_timestamp} for item: \${item_name}(id \${item_id}) x \${quantity}, total \${price}. The shipment is \${order_status}"
3. Selection: Based on the message, identify the item(s) that the customer is referencing. Please bear in mind that the customer might be referring to multiple items.
4. Formulate JSON Object: Provide your results in a JSON object structured as:
{
"item_id": "<item_id for referenced item>",
"item_name": "<item_name for referenced item",
"order_id": "<order_id for referenced item>",
"quantity": "<quantity for referenced item>"
}
5. Respond with the final JSON object, do not include any other text or comments.
Example response:
[
{
"item_id": "ITEM12345",
"item_name": "Example Product A",
"order_id": "ORD012345",
"quantity": 1
},
{
"item_id": "ITEM52365",
"item_name": "Example Product B",
"order_id": "ORD013378",
"quantity": 2
}
]
---Start of Message---
Subject: Need help
I hope this email finds you well. My name is Jake, and I am writing to request an update on the shipment status of my recent order.
---End of Message---
---Start of Items---
Order Number 20231115-001 placed on 2023-11-15T10:45:00 for item: Stainless Steel Water Bottle (id 987) x 1, total $15.99. The shipment is not fulfilled.
Order Number #20231114-045 placed on 2023-11-14T14:30:00 for items: Bluetooth Headphones (id 456) x 1, total $89.99, Portable Charger (id 321) x 2, total $39.98. The shipment is in-transit.
Order Number #20231113-123 placed on 2023-11-13T08:15:00 for items: Wireless Mouse (id 789) x 1, total $24.99, USB-C Cable (id 654) x 3, total $29.97, Laptop Sleeve (id 532) x 1, total $19.99. The shipment is delivered.
---End of Items---Response
[
{
"item_id": "987",
"item_name": "Stainless Steel Water Bottle",
"order_id": "20231115-001",
"quantity": 1
},
{
"item_id": "456",
"item_name": "Bluetooth Headphones",
"order_id": "20231114-045",
"quantity": 1
},
{
"item_id": "321",
"item_name": "Portable Charger",
"order_id": "20231114-045",
"quantity": 2
},
{
"item_id": "789",
"item_name": "Wireless Mouse",
"order_id": "20231113-123",
"quantity": 1
},
{
"item_id": "654",
"item_name": "USB-C Cable",
"order_id": "20231113-123",
"quantity": 3
},
{
"item_id": "532",
"item_name": "Laptop Sleeve",
"order_id": "20231113-123",
"quantity": 1
}
]Results
👎Agent must correctly identify all items referenced by the customer.
👍Agent must respond in a pre-defined format.
It did great in adhering to the format but was not able to associate the customer’s request for “my recent order”. The AI is taking a broader view of what recent means, and is biasing to error on the side of selecting too much vs too little. However, this highlights a gap in understanding compared to a human agent.
Round 2
I tightened interpretation of “recent” with explicit instructions on how to handle “singular terms” in Step 3.
In cases where the customer uses singular terms to refer to their order, such as 'my order', 'the order', or 'my recent order', prioritize identifying only one order that most closely matches the description. If multiple orders could fit but the customer's reference is singular, select the most recent or the closest to completion order.Prompt
Your Role: As an expert customer service agent, your primary task is to interpret customer emails and deduce the referenced items from the details given, creating a response in the form of JSON objects.
Instructions:
1. Analyze the Message: Carefully read the provided customer message. Read the entire message before proceeding.
2. Analyze available Item options: Items that the customer ordered will be presented in this format.
"Order Number \${order_id} placed on \${placed_timestamp} for item: \${item_name}(id \${item_id}) x \${quantity}, total \${price}. The shipment is \${order_status}"
3. Selection: Based on the message, identify the item(s) that the customer is referencing. Please bear in mind that the customer might be referring to multiple items.
In cases where the customer uses singular terms to refer to their order, such as 'my order', 'the order', or 'my recent order', prioritize identifying only one order that most closely matches the description. If multiple orders could fit but the customer's reference is singular, select the most recent or the closest to completion order.
4. Formulate JSON Object: Provide your results in a JSON object structured as:
{
"item_id": "<item_id for referenced item>",
"item_name": "<item_name for referenced item",
"order_id": "<order_id for referenced item>",
"quantity": "<quantity for referenced item>"
}
5. Respond with the final JSON object, do not include any other text or comments.
Example response:
[
{
"item_id": "ITEM12345",
"item_name": "Example Product A",
"order_id": "ORD012345",
"quantity": 1
},
{
"item_id": "ITEM52365",
"item_name": "Example Product B",
"order_id": "ORD013378",
"quantity": 2
}
]
---Start of Message---
Subject: Need help
I hope this email finds you well. My name is Jake, and I am writing to request an update on the shipment status of my recent order.
---End of Message---
---Start of Items---
Order Number 20231115-001 placed on 2023-11-15T10:45:00 for item: Stainless Steel Water Bottle (id 987) x 1, total $15.99. The shipment is not fulfilled.
Order Number #20231114-045 placed on 2023-11-14T14:30:00 for items: Bluetooth Headphones (id 456) x 1, total $89.99, Portable Charger (id 321) x 2, total $39.98. The shipment is in-transit.
Order Number #20231113-123 placed on 2023-11-13T08:15:00 for items: Wireless Mouse (id 789) x 1, total $24.99, USB-C Cable (id 654) x 3, total $29.97, Laptop Sleeve (id 532) x 1, total $19.99. The shipment is delivered.
---End of Items---
Response
[
{
"item_id": "987",
"item_name": "Stainless Steel Water Bottle",
"order_id": "20231115-001",
"quantity": 1
}
]Results
👍Agent must correctly identify all items referenced by the customer.
👍Agent must respond in a pre-defined format.
Got both parts right. I am concerned that the “singular terms” instruction might bias the prompt to this exact message. We will check for that in our validation tests.
Validation
I tested this prompt across a series of actual customer messages. Below are a few examples with sanitized data.
1. Re: A shipment from order #20231113-123 has been delivered
Hi,
It says my order was delivered on 11/15 but I have not received it and it has been a few business days since then. Please help!
All the best,
JackResult 👍
[
{
"item_id": "789",
"item_name": "Wireless Mouse",
"order_id": "20231113-123",
"quantity": "1"
},
{
"item_id": "654",
"item_name": "USB-C Cable",
"order_id": "20231113-123",
"quantity": "3"
},
{
"item_id": "532",
"item_name": "Laptop Sleeve",
"order_id": "20231113-123",
"quantity": "1"
}
]2. Missing item
Hello,
I received the large box of packets that I ordered, but the cable was not included in the shipment. Is it still being sent?
Thank you,
JackResult 👍
[
{
"item_id": "654",
"item_name": "USB-C Cable",
"order_id": "#20231113-123",
"quantity": 3
}
]3. Request exchange or refund
Hi, I did not realize the subscription was set as monthly. I am not ready for the next shipment to come in yet. Please cancel and refund order my last order
Thank you,
JackResult 👍
[
{
"item_id": "987",
"item_name": "Stainless Steel Water Bottle",
"order_id": "20231115-001",
"quantity": 1
}
]Conclusion
Our AI agent seems ready to tackle item selection. A key learning was understanding how “recent” was interpreted by default and how we modified that understanding for e-commerce support.
AI Powered Operations with MelodyArc CX
MelodyArc CX is the world's first all-in-one customer operations platform, reducing operating costs by >50%. It's a single end-to-end solution that handles daily customer operations by providing both agents and AI together in one powerful platform.
MelodyArc CX layers on top of your existing CRM and other customer service channels to manage all the moving parts. From responding to and resolving customer requests, to implementing and cascading policies, to forecasting and analytics, MelodyArc CX takes on daily operations. It's more cost effective than agents alone and ensures higher resolution rates than automation.
Pay-as-you-go pricing with no minimum commitment means MelodyArc flexes with your business.
Book a demo today to learn more.