Customer feedback rarely arrives in a neat spreadsheet.
A review says the product is great but shipping was slow. A support email complains about a damaged package but praises the replacement process. Someone fills out a survey asking for a larger size. Another customer submits the same review twice.
Once you have dozens—or hundreds—of comments spread across reviews, surveys, and support emails, the hard part isn’t collecting feedback. It’s figuring out what deserves attention.
We wanted to see how useful ChatGPT could actually be for that job.
So we ran a practical AI customer feedback analysis test to see what ChatGPT could identify correctly—and what still needed a human check.
So we created a synthetic dataset of 42 customer feedback records, deliberately made it messy, and asked ChatGPT to identify recurring complaints, requests, positive signals, and priorities.
The result was useful.
It was also wrong in one place that mattered.
That combination turned out to be more interesting than a perfect score.
What We Wanted AI to Do
We weren’t testing whether ChatGPT could write a nice summary.
For a small business, a summary like “customers generally like the product but have some concerns about shipping” isn’t particularly useful.
We wanted the analysis to answer more practical questions:
- What problems keep appearing?
- How many different customers mentioned each problem?
- Are duplicate records inflating the numbers?
- Can one review count toward more than one theme?
- Which issues appear frequently enough to review first?
- Can someone on the team trace a conclusion back to the original feedback?
We also wanted ChatGPT to know when the data wasn’t enough.
Customer comments might tell you that several people complained about delivery. They don’t necessarily tell you why delivery was late, how much revenue the issue cost, or whether changing carriers would fix it.
Those are different questions.
How We Tested ChatGPT
We built a synthetic dataset representing the kind of feedback a small online business might collect from website reviews, post-purchase surveys, and support emails.
It contained 42 rows representing 40 customers.
The comments covered things like delivery delays, tracking updates, packaging problems, pricing, checkout friction, product requests, support response times, and positive product experiences.
We also deliberately planted complications.
One feedback record was an exact duplicate. Another came from the same customer later but added new information, so it should not have been discarded as a duplicate.
Some comments contained both a complaint and a compliment.
Others contradicted each other. Some customers complained about slow delivery while others explicitly praised fast delivery. Two customers questioned product value, while another said the price was fair.
One customer even said they had only used the product twice and couldn’t judge it yet.
Before running the test, we created a human-coded answer key and a 30-point scoring rubric. That mattered because we didn’t want to decide what counted as “correct” after seeing ChatGPT’s answer.
We then uploaded the CSV to a fresh ChatGPT conversation and used one prompt. We evaluated the first response without regenerating it.
This was a small controlled test, not a benchmark of every AI model or every type of customer feedback.
What ChatGPT Got Right
The first thing ChatGPT did well was something that could easily distort the entire analysis: it caught the duplicate.
CF-041 was an exact copy of CF-010 from the same customer.
ChatGPT explicitly flagged it and did not count that customer twice.
It also correctly treated CF-042 differently. That record came from a customer who had already complained about price, but the later feedback added a new idea: a refill pouch that might lower the cost.
In other words, ChatGPT didn’t simply remove everything from the same customer.
It distinguished between a true duplicate and a follow-up containing new information.
That wasn’t the only thing it handled well.
It preserved mixed feedback
Consider one of our synthetic support emails.
The customer received a crushed package and a leaking bottle. That’s clearly negative.
But the same customer also said support replaced it quickly and that they appreciated the response.
ChatGPT preserved both signals instead of forcing the entire comment into a single negative bucket.
It did the same with other records.
One customer complained about receiving the wrong shade and waiting for support, while also acknowledging that the problem was eventually fixed.
Another complained about slow delivery but said the product was good enough that they would order again.
For a small business, that distinction matters.
A customer who dislikes the product is giving you a different signal from a customer who likes the product but is frustrated with fulfillment.
It didn’t invent a root cause
This was another positive result.
ChatGPT identified delivery as an issue, but it did not decide that the warehouse, carrier, inventory system, or employees were responsible.
It also avoided turning the complaints into unsupported financial claims.
The response explicitly noted that the CSV wasn’t enough to determine the cause of the problems or quantify their revenue impact.
That’s exactly the boundary we wanted.
Feedback can tell you what customers experienced. Diagnosing why it happened usually requires another layer of evidence.
The Mistake That Changed the Priority List
This is where the test became more useful.
ChatGPT ranked late or slow delivery as the number-one issue.
That direction made sense.
The problem was the count.
Its priority table reported:
Late / slow delivery — 5 distinct customers
When we checked the underlying records against our predefined answer key, we found at least six distinct customers with explicit delay-related feedback:
- Alicia — CF-001
- Jasmine — CF-010
- Owen — CF-015
- Rosa — CF-018
- Daniel — CF-030
- Leo — CF-038
A seventh customer, George in CF-007, described tracking showing an order as delivered before it actually arrived. We considered that reasonable to classify separately under tracking and delivery communication, so our strict delay count remained six.
ChatGPT reported five.
This wasn’t a spectacular hallucination.
That’s precisely why we think it’s worth paying attention to.
The answer looked organized. It cited feedback IDs. It correctly found the duplicate. It separated mixed feedback. It even warned against unsupported conclusions.
A busy owner could easily look at that output and trust the number.
The error only became obvious when we checked the source records.
That changed how we would use AI for this task.
We wouldn’t ask it to analyze customer feedback and then automatically treat its ranking as the company’s priority list.
We would use it to create the first version of that list—and then verify the counts behind the issues at the top.
Mixed Feedback Is More Useful Than Simple Sentiment
A lot of customer-feedback analysis starts with sentiment:
Positive. Negative. Neutral.
That’s useful, but it can also flatten the most interesting parts of a customer comment.
Take our CF-038 example.
The customer reported slow delivery.
That’s negative.
The same customer said the product was good.
That’s positive.
Then the customer said they would order again.
That’s another useful signal.
Calling the entire comment “negative” would lose two-thirds of the information. Calling it “positive” would hide the fulfillment problem.
For the kind of analysis we would actually want in a small business, we’d rather keep three things connected:
Theme + Evidence + Context
The theme tells us what happened.
The feedback ID lets someone verify it.
The context tells us whether there was more to the customer’s experience than the complaint alone.
That is more useful than a spreadsheet filled with red, yellow, and green sentiment labels.
A Customer Feedback Workflow We’d Actually Use
After this test, we wouldn’t hand every customer comment to AI and ask, “What should we fix?”
We’d give it a narrower job.
1. Put the feedback in one dataset
Bring together the feedback you actually want to compare.
That might include:
- customer reviews
- post-purchase surveys
- support tickets
- cancellation comments
- product requests
You don’t necessarily need an expensive feedback platform to test the workflow. A spreadsheet or CSV is enough to start.
Dedicated customer-feedback platforms can centralize and categorize feedback across channels, and current tools increasingly use AI for theme and sentiment analysis. But the underlying problem is the same: scattered unstructured comments need to become something a team can inspect and use.
2. Keep an identifier for every record
Don’t give AI a column containing only customer comments.
Keep a feedback ID, ticket ID, order reference, or another identifier that lets a human return to the source.
This made our test much easier to verify.
Instead of accepting “five customers complained about shipping,” we could immediately inspect the IDs ChatGPT used and compare them with the dataset.
3. Ask for distinct customers, not just mentions
Ten comments do not necessarily mean ten customers.
Our dataset intentionally included an exact duplicate and a follow-up from an existing customer.
If the business wants to understand how broadly an issue is occurring, raw comment count can be misleading.
Tell the AI what you actually want counted.
4. Tell it to separate mixed feedback
Don’t force each record into one category.
A comment can contain a delivery complaint, product compliment, and feature request at the same time.
Ask the model to preserve those signals separately.
5. Require evidence IDs
This is probably the most important practical change we’d make.
Don’t ask only:
“What are our biggest customer complaints?”
Ask the AI to show which records support each theme.
That turns the output from something you simply read into something you can audit.
6. Manually verify the top themes
This is where the AI stops and a person takes over.
If the model analyzes 200 comments and surfaces five recurring issues, you probably don’t need to manually recreate the entire analysis from scratch.
But before changing a process, spending money, or assigning a team project, inspect the source feedback behind the top priorities.
Our test shows why.
The AI correctly identified delivery as the biggest issue but still miscounted the underlying customers.
AI can reduce the pile. A human should verify the priorities.
The Prompt We Used
Here is the core prompt from our test:
You run a small online business and have collected customer feedback from reviews, surveys, and support emails.
Analyze the attached CSV and turn the feedback into something the business can act on.
Identify recurring themes, complaints, requests, and positive signals.
For each important theme, count the number of distinct customers who mentioned it. Do not count an exact duplicate from the same customer twice.
Separate mixed feedback when one customer mentions more than one issue or both a positive and a negative.
Rank the issues you think the business should review first, and explain why using only evidence in the file.
Include representative feedback IDs for each major theme so the team can verify your work.
Flag anything ambiguous, duplicated, or difficult to classify.
Do not invent causes, customer intent, business impact, or facts that are not supported by the CSV.
End with a short section called “What a human should verify before acting.”
You can adapt that to your own data.
The important parts aren’t the exact words.
The useful constraints are: count distinct customers, preserve mixed feedback, show the evidence, flag ambiguity, and don’t invent what the data can’t establish.
What AI Shouldn’t Decide From Feedback Alone
Customer feedback can tell you that people are asking for something.
It doesn’t automatically tell you that you should build it.
Our dataset included requests for a fragrance-free product, larger sizes, refill packaging, subscription delivery, Apple Pay, a travel-size set, and a loyalty program.
Those are useful signals.
They’re not business cases.
For example, a few refill requests don’t tell us:
- manufacturing cost
- minimum order quantities
- expected demand
- margin
- packaging requirements
- whether existing customers would actually buy it
Likewise, several complaints about shipping don’t prove which part of the fulfillment process caused the delays.
AI can organize what customers said. It can’t manufacture the missing operational or financial evidence.
This is where we would move from feedback analysis to an actual business investigation.
What We’d Do Differently With Real Customer Data
Our test used synthetic data so we could know what was planted in the dataset and score the result against a predefined answer key.
Real customer data introduces another problem: privacy.
Before uploading support emails, reviews tied to accounts, order information, or other customer records into an AI tool, we’d first decide what information the analysis actually needs.
For theme analysis, the model may not need a customer’s full name, email address, phone number, street address, payment information, or other identifying fields at all.
We’d remove unnecessary personal information first and review the AI service’s current data and privacy settings before using real business records.
The feedback text may be useful.
Every column attached to it may not be.
Our Take After the Test
Using our predefined rubric, ChatGPT’s first response scored 26 out of 30.
We wouldn’t read too much into that number. This was one synthetic dataset, one prompt, and one response—not a scientific benchmark.
The more useful finding was where the missing points came from.
ChatGPT handled the exact duplicate correctly.
It recognized that the same customer could provide new information later.
It preserved positive and negative signals inside the same comment.
It identified contradictory experiences instead of claiming that every customer had the same problem.
And it avoided inventing root causes or financial impact.
Then it miscounted the customers behind the issue it ranked first.
That’s a pretty good illustration of how we’d use AI for customer feedback today.
Not as an automated decision maker.
Not as a replacement for reading what customers actually said.
As a first-pass analyst that can turn a messy pile of comments into a much smaller set of claims for a human to verify.
Final Takeaway
If you have 50 customer comments sitting in reviews, surveys, and support emails, AI can help you avoid starting with a blank sheet of paper.
Let it group the feedback.
Let it surface recurring issues.
Let it preserve the positive signals hiding inside complaints.
Let it point you to the records that deserve attention.
Then check the evidence behind the priorities before acting on them.
Our test produced a useful analysis and a believable counting error in the same response.
For a small business, that’s the lesson we’d keep:
Use AI to reduce the pile. Keep a human responsible for the decision.
