Marginally Better S01E04 When Customer Experience Goes Wrong: AI Promises, Survey Fatigue, and Subscription Traps

July 2, 2025
Marginally Better, Podcast
18 min read

AI gone rogue, broken feedback loops, and cancellation nightmares—this episode of Marginally Better exposes how even well-intentioned CX efforts can backfire. Joe Taylor, Jr. unpacks why 64% of customers want AI out of customer service, how survey fatigue is killing Net Promoter Scores, and why the subscription economy may finally be forced to clean up its act. From chatbot fails to $881 million algorithm mistakes, this episode explores what happens when technology outpaces empathy—and what brands must do to regain trust.

Episode Links:

Additional Resources:

Transcript:

[00:00:00] Announcer: From the global headquarters of Johns & Taylor in beautiful New Jersey, it’s Marginally Better. Here’s your host, Joe Taylor, Jr.  

[00:00:13] Joe Taylor, Jr.: On the show this week, 64% of customers would prefer companies avoid using AI in customer service entirely. And yet, businesses keep deploying chatbots, often with unexpected results. We’ll explore what happens when artificial intelligence starts making up company policies. Why Instagram users revolted against AI search and how Zillow lost a bunch of real money betting on algorithms. We’ll also dive into the collapse of America’s favorite business metric, and we’ll uncover the subscription economies dirty little secret. That’s all coming up after the break on Marginally Better. 

[00:01:07] Welcome to Marginally Better, a show about business innovation and the American economy. I’m Joe Taylor, Jr.  

[00:01:15] There’s a moment that happens in every boardroom across America. Some executive leans back in a chair, looks at the quarterly budget and says, we need to improve customer service, but we also need to cut costs. How about AI? And that’s when things get interesting; not always good interesting.  

[00:01:38] Story one the chat-bot that made its own rules. Let’s take what happened at a company called Cursor AI. Their customer service chat chatbot didn’t just answer questions, Ii started creating company policies out of thin air. 

[00:01:55] A customer asked about device limitations and, instead of saying I don’t know or let me connect you with a human, the AI just invented a policy made up rules that didn’t exist. The results: angry customers, subscription, cancellations, and a very awkward all hands meeting where somebody had to explain how artificial intelligence had become an artificial executive. 

[00:02:22] Here’s the thing, according to research from the Nielsen Norman Group, the folks who literally wrote the book on user experience, AI must solve real problems, not just implemented for novelty. But that’s exactly what we’re seeing in cases like this. Companies chasing shiny technology and not really fixing underlying customer pain points. 

[00:02:47] Story two, when Instagram broke its own search. Speaking of fixing things that weren’t broken, let’s talk about Instagram. Last year, Meta decided to replace Instagram’s perfectly functional search feature with an AI chat interface. Users who just wanted to find a specific post or an account suddenly have to have a conversation with a bot. 

[00:03:12] The backlash was swift and merciless. Users didn’t want to chat with AI, they wanted to search. Meta quietly rolled back the change in most places, but you can still see that AI prompt in a lot of places on Facebook and Instagram. They didn’t do it quickly enough to avoid infuriating a bunch of their users by blocking useful functionality for unnecessary AI features. 

[00:03:41] Story three, the $881 million algorithm mistake. Sometimes the stakes are a lot higher than just annoyed users. Some companies bet the farm on new technology and lose spectacularly. Just ask anyone you know working at Zillow.  

[00:04:03] The real estate giant spent years developing an artificial intelligence system called Zestimate to predict home values. You’ve seen this in the app. You probably get emails if you own a home or if you’ve ever shopped for a home about what home values might be in a particular market, all the way down to the block. Now Zillow’s very confident about this artificial intelligence. And they got so confident that instead of just recommending a price point to a potential buyer or seller, they started using the same tool to determine which properties to purchase as they started buying and flipping houses themselves. 

[00:04:49] The AI was wrong – consistently, catastrophically wrong. Zillow reported that they ended up losing more than $881 million. They ended up laying off a quarter of their workforce. It turns out that predicting human behavior and market dynamics is more complicated than training an algorithm on historical data. 

[00:05:18] Story four, the empathy gap. Here’s what’s really fascinating. While companies are rushing to deploy AI in customer service, customers are running in the opposite direction. A 2024 Gartner study found that 64% of customers would prefer companies avoid using AI in customer service entirely. That’s because 67% of consumers say traits like creativity, empathy, and friendlines – you know, human traits – they’re important, they lead to better outcomes. There’s something about talking to a machine that just doesn’t satisfy our need for connection when we’re frustrated or confused.  

[00:06:02] Steve Blood from Five9, a customer experience company, puts it bluntly. Quote, “If your provider can’t offer you a route for assisted service, stop investing because it’s a failed strategy.”  

[00:06:17] Story five, the integration reality check. Then there’s the technical reality that nobody talks about in those boardroom presentations. Many companies have what experts diplomatically call tightly integrated contact center solutions. Translation, their systems are held together with digital duct tape and possibly a lot of prayer. 

[00:06:41] Bringing AI into these environments requires careful planning, extensive testing, and usually a complete overhaul of existing processes. That $50,000 AI chatbot license suddenly becomes a $500,000 systems integration project. McKinsey studied this and found that while organizations see an average 3 dollar and 70 cent return for every dollar invested generative AI, only 1% of companies believe that they are at AI maturity. The other 99% are still figuring it out often at their customers expense. And McKinsey said this, okay?  

[00:07:27] The lesson here isn’t that AI is bad, it’s that implementing technology without understanding your customer’s actual needs is a recipe for expensive mistakes. When you lead with technology instead of customer problems, you end up with solutions looking for problems, not the other way around. 

[00:08:01] After the break, what happens to Net Promoter? That’s coming up on Marginally Better. 

[00:08:12] Johns & Taylor Ad: You know what drives me crazy, beautiful websites that don’t actually work. I’m Joe Taylor, Jr. I see this all the time in my work at our user experience agency, Johns & Taylor. Here’s what I’ve learned after more than 20 years of building websites, most organizations approach this backwards. They jump straight into pretty mockups or use DIY site builders without understanding what their site actually needs to accomplish. 

[00:08:37] That’s why my team and I start with strategy, not just aesthetics. We’ve cracked the code on building websites that are both beautiful and functional. And right now we’re offering complimentary website and user experience assessments for business podcast listeners. No sales pitch, just honest feedback about what’s working on your site and what isn’t. 

[00:08:58] If you are tired of a website that looks good but doesn’t deliver results, visit makethewebsiteworkforme.com. Discover what your website could really accomplish. Makethewebsiteworkforme.com. 

[00:09:17] Joe Taylor, Jr.: It is Marginally Better. I’m Joe Taylor, Jr.  

[00:09:19] I am and have been for a long time, a big Net Promoter nerd. I think that the underlying ethos of Net Promoter is a really great way for managers to think about how to continually improve their systems. But something’s not quite right lately. And to tell this story, our team and I are gonna tell you a story as a parable. We’re gonna make up a couple of personas to be able to tell this effectively.  

[00:09:50] So I’m gonna start by telling you about someone that we’ll call Sarah. She is not a real person. She represents thousands of customer experience executives, like the folks that we talked to at Johns & Taylor. We are all having the same nightmare in some manner right now. 

[00:10:08] Sarah built her career on data, specifically on a single number; the Net Promoter score. For 15 years NPS was her North Star. On a scale of 0 to 10, how likely are you to recommend our company to a friend or a colleague. Simple question. What a clean metric. Executive dashboard goal. Sarah’s NPS program was beautiful. Monthly surveys to 10,000 customers. Automated email campaigns, color-coded scorecards that went right into the C-suite. She could tell you their score down to the decimal point on any given day. 47.3. Not great, trending upward until. It wasn’t.  

[00:10:59] First, the response rates started dropping what used to be a 40% response rate, fell to 25%, then 15, then single digits. Sarah’s team tried everything. They ab tested some email subject lines. They added gift card incentives. They made the survey shorter. Nothing worked and then came the data that broke Sarah’s world. She discovered research showing that survey response rates have collapsed across every industry. The Pew Research Center reported that public opinion survey responses have plummeted from 36% in 1997 to just 6% by 2018. 

[00:11:49] The US Department of Defense saw their active-duty survey response rates dropped from 40% to 15% between 2004 and 2018. And that’s not the worst part. The worst part was what Sarah learned about the customers who were still responding to surveys.  

[00:12:09] It turns out as of 2025, only certain types of people are still filling out customer satisfaction surveys. They are the extremely happy customers who love talking about their experience and the extremely angry customers who just want to vent. And almost nobody in between. Sarah wasn’t measuring customers satisfaction anymore. She’s measuring the opinions of statistical outliers.  

[00:12:44] Sarah’s data scientist, we’re gonna call this person Marcus. Brilliant. 28 years old. Still got a lot of growth to do in this role, somebody that I love coaching. But Marcus dropped the bomb. No tact in the meeting when this happened. He’d been digging into the research behind NPS and what he found, it shook Sarah to her core predictive validity is garbage. Studies show NPS actually performs worse than basic satisfaction metrics in predicting actual business growth. Plus there’s massive cultural bias. Customers in the Netherlands consistently rate companies lower than Americans, and that makes global comparisons meaningless.  

[00:13:34] So even Fred Reichheld, someone who’s credited with inventing NPS, he’s saying that companies are misusing it. He calls it a tragedy of the commons. 

[00:13:45] When organizations tie NPS scores to employee compensation, as many companies now do, it makes staff focus on gaming the rating instead of actually helping customers.  

[00:13:59] Think about Sarah – and you might be Sarah, you might very much feel what Sarah’s feeling in this conversation. The metric that she had spent most of the past decade optimizing, it wasn’t only broken, it was possibly making her customer service worse. 

[00:14:18] But Marcus had a little more cultural research to show Sarah. And we’re living in what one researcher calls ‘The Era of Meh.’ When everything becomes automated, when AI can create good enough content and experiences, when processes are optimized for efficiency instead of human connection, customers stop caring enough to respond thoughtfully to anything. 

[00:14:49] 74% of customers will only answer five questions or less In surveys, 70% of people abandon surveys before they finish them. Only 9% take time to answer long surveys thoughtfully.  

[00:15:07] So Sarah thinks about her beautiful 15 question NPS survey. The one that used to ask about everything from product quality to brand perception. It used to take eight minutes to complete. Customers, they’re abandoning it in droves.  

[00:15:26] But you need the data, right? I mean, how do you make decisions without customer feedback? And in this case, we tell our clients it’s time to stop asking and start watching. Look at real customer behavior data. Look at the purchase patterns, look at your support tickets, what social media sentiment looked like. Churn predictions based on actual actions, not stated intentions.  

[00:15:57] So look, Marcus said points to a graph in the report. This customer says they’d recommend us to a friend to give us a 9 out of 10, but they haven’t made a purchase in six months. They just downgraded their subscription. Their behavior tells us a lot more than their survey response ever could. 

[00:16:18] So what do you do in this situation? It’s time to listen differently. A lot of different ways to approach it now. Customer advisory boards can give you deep insights. Behavioral analytics can highlight trends. Social media monitoring can give you some real-time sentiment. Even listening in on your support calls, the analysis there can highlight the pain points in the process. Measure what matters; retention, lifetime value, organic growth.  

[00:16:49] The hardest part for a lot of folks is not abandoning NPS. And to some degree you don’t necessarily have to abandon it all the way. But the hardest part for many of us who used to lean on NPS is to admit that in your quest to quantify customer satisfaction, you stopped actually listening to customers. The surveys themselves have often become a barrier between companies and real customer insights.  

[00:17:18] So Sarah prepared a presentation for the executive team and she felt something she hadn’t experienced in years, confidence that her data actually meant something. She was finally ready to tell her company what their customers were really thinking and not just what the few willing to take surveys were saying. 

[00:17:40] The age of survey fatigue is ending, but the age of authentic customer understanding is just beginning.  

[00:17:51] After the break, we’re finally gonna get around to canceling that meditation app we’ve been meaning to get rid of. It’s Marginally Better. 

[00:18:06] Johns & Taylor Ad: If you are a business owner,  

[00:18:08] I need to tell you something that might sting. Your beautiful website is probably costing you thousands in lost revenue. The Nielsen Norman Group found that 76% of professional websites sites with million dollar budgets perform mediocre. To pour on conversion, and if you built your site on Squarespace, Wix, or GoDaddy, your conversion rate is probably half of what it should be. Research shows the average business website can improve conversions by 35% or more just by fixing a few basic problems.  

[00:18:42] I’m launching a new podcast called Website Reality Check; 10 episodes each one walking you through research backed fixes. You can implement yourself even on DIY platforms. Get free access to the first episode at websiterealitycheck.com. 

[00:18:59] Stop leaving money on the table with websites that look great, but don’t convert. That’s websiterealitycheck.com.  

[00:19:09] Joe Taylor, Jr.: It’s Marginally Better. I’m Joe Taylor, Jr.  

[00:19:11] And I’d like you to picture a scene. You’re sitting on your couch, it’s 11:00 PM you’ve had a long day, and you decide to try that meditation app everyone’s been talking about. Seven days free, cancel any time according to the ad. The famous last words. You download the app, you enter the email, then comes the payment screen. And here’s where things get interesting. The most prominent button says $9.99 per month. That’s reasonable. Sure. Buried in tiny texts below that, they mention a $49 annual wellness fee. Well, that wasn’t in the commercial, but you’re tired. You need to meditate. You tap through. What’s 49 bucks once a year, right?  

[00:19:59] So you fast forward three months, you check your bank statement, there’s the $9.99 again and again. You’ve used that app twice since downloading it. But hey, you’re building a healthy habit. That’s fine. You get to month six and decide, you know, I really should just cancel that thing. How hard can it be?  

[00:20:20] Welcome to what Consumer Protection experts call the Roach Motel pattern. It’s easy to check in, nearly impossible to check out. First, you can’t find a cancel button anywhere in the app. 20 minutes of searching through menus leads you to a tiny managed subscription link and that takes you to a website. And now the website informs you that since you signed up through the app, you need to cancel through your phone’s app store. And the app store, maybe it sends you back to the app. The app sends you back to the website. It’s digital ping pong, but no one ever wins. And finally you find a phone number, but they’re only open weekdays, 9 – 5 Pacific time… because people only apparently want to cancel subscriptions during West Coast business hours. Well, okay. You finally get a human on the phone. They’re very sorry to hear that you want to cancel. They’re very sorry. They’d like to help you succeed with your meditation practice. Have you tried the sleep stories? What about the anxiety course? And I just want to cancel you say. I understand replies the impossibly cheerful customer service rep. But first, can I offer you three months free if you stick with us?  

[00:21:40] Now, this is a very common scenario. The Federal Trade Commission just sued Uber for putting users through as many as 23 screens and up to 32 separate actions just to cancel their Uber One subscription. And that’s a bummer because I use Uber One and I like it. I hadn’t even thought about canceling that, but Ouch. 23 Screens, somebody there literally designed a process that would make anybody just give up trying to cancel. 

[00:22:14] It’s not just Uber. The subscription economy is now worth $1.5 trillion and growing. The average American spends $273 per month on subscriptions. But here’s the kicker. When you ask, if you ask folks, they believe they only spend closer to $86 per month.  

[00:22:38] Now we’re all living in a subscription fog. We’ve all paid for services that we have forgotten about. Planet Fitness has turned this into an art form. I go to Planet Fitness maybe three times a year. And I will tell you I do it because I love the little massage chairs they have. You sign up online, it takes two minutes. But to cancel, you need to visit your home gym in person, or you need to send a certified letter. A certified letter in 2025 to cancel a gym membership. The last time I went in to cancel, I ended up just sitting in the massage chair again and figured it was less of a hassle to just let it go and occasionally show up and do that, and maybe I will get fit next year.  

[00:23:29] Try to cancel early from a typical gym membership 12 month contract. That’s gonna be a $58 buyout fee. Plus an annual fee that they mentioned in passing during the signup.  

[00:23:45] Adobe may even be more creative than some of the gyms that are out there. Their most popular plan looks like it costs $2.99 cents per month, but it’s actually an annual plan paid monthly. If you wanna cancel before the year is up, you still owe all of your remaining payments. Now, for some customers, that could be $200 that you have to pay just to stop using your software.  

[00:24:12] The pattern is everywhere. Netflix makes you confirm that you really want to give up award-winning shows. Amazon’s cancellation process used to be so complicated that in internal documents they called it the Iliad, named after Homer’s epic poem about a journey that takes 10 years to complete. 

[00:24:32] Customers, they’re fighting back against this stuff, and increasingly they’re winning. The Federal Trade Commission just implemented something called the Click to Cancel rule. Starting July, companies have to make canceling as easy as signing up. If you can subscribe online, they have to let you cancel online. If it takes one click to sign up, it can only take one click to cancel. And states are getting aggressive about this, too. California now requires that if you sign up online, you must be able to cancel online. No phone calls, no letters, no visits to your home gym.  

[00:25:11] Credit card processors are even getting involved. MasterCard and Visa now require clearer subscription disclosures and easier cancellation processes. But the real victory is happening in boardrooms across America where executives are finally realizing that trapping customers creates enemies, not loyalty. When 25% of Americans say they’ll never return to a brand after a frustrating experience and 47% abandoned purchases due to unexpected fees, the math becomes pretty simple.  

[00:25:45] Companies that make it easy to leave are discovering something counterintuitive. Customers are more likely to stay. The subscription trap era is ending. Not because companies suddenly grow a conscience, but because customers have had enough, regulators are paying attention, and economic costs of being deceptive are finally higher than the benefits. Your Move meditation app. 

[00:26:14] Thanks for listening to Marginally Better. If you like what you heard, please help us out. Leave a quick review on Apple Podcasts. It will help us spread the word about the show to people like you who care deeply about great customer experiences. And if you wanna get behind the scenes notes from me and the rest of the team, including links to all of the data that we cite throughout the show, go to marginallybettershow.com or follow the link in our show notes. 

[00:26:42] Marginally Better is a Calufrax radio production. Our producer is Nicole Hubbard with research by Connie Evans. I’m Joe Taylor, Jr. 

After a decade in broadcast media, Joe developed early online platforms for NPR, PBS, and AOL. Today, he helps our clients tell compelling brand stories through audio, visuals, and software.

You may also like

Get Expert Website Help